2023-10-21T03:52:10.227 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-21T03:52:10.229 DEBUG:teuthology.run:Teuthology command: teuthology --name rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi --description rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/defaults msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} --verbose --owner scheduled_rraja@teuthology --archive /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433375 -- /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433375/orig.config.yaml 2023-10-21T03:52:10.265 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T03:52:10.348 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433375 branch: ajarr-wip-extra-debug-62994-oct-19 description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/defaults msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7433375' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: ajarr-wip-extra-debug-62994-oct-19 ceph: conf: client: rbd default data pool: datapool rbd default features: 61 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: ajarr-wip-extra-debug-62994-oct-19 sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab owner: scheduled_rraja@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1449 sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab sleep_before_teardown: 0 suite: rbd:cli suite_branch: ajarr-wip-extra-debug-62994-oct-19 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab targets: smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGxiak4qSskTc2wF5c0efzOeCTK+1QFx5H7sKK9+dm0dquM9GdwhHn9zL1QnphPwznKmEs7wAhHBjtwlgK4pd24= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create datapool 4 - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-20_15:15:00 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-21T03:52:10.348 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa; will attempt to use it 2023-10-21T03:52:10.349 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks 2023-10-21T03:52:10.349 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-21T03:52:10.352 INFO:teuthology.task.internal:Checking packages... 2023-10-21T03:52:10.376 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab' 2023-10-21T03:52:10.376 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-21T03:52:10.376 INFO:teuthology.packaging:ref: None 2023-10-21T03:52:10.377 INFO:teuthology.packaging:tag: None 2023-10-21T03:52:10.377 INFO:teuthology.packaging:branch: ajarr-wip-extra-debug-62994-oct-19 2023-10-21T03:52:10.377 INFO:teuthology.packaging:sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T03:52:10.378 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=ajarr-wip-extra-debug-62994-oct-19 2023-10-21T03:52:10.521 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T03:52:10.522 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-21T03:52:10.529 INFO:teuthology.task.internal:no buildpackages task found 2023-10-21T03:52:10.529 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-21T03:52:10.552 INFO:teuthology.task.internal:Saving configuration 2023-10-21T03:52:10.566 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-21T03:52:10.572 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-21T03:52:10.598 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433375', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-21 03:46:35.226366', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBECFuMqQrxfvjVPGKZ/WysYkLWld4AsSVJvY0z/wX18'} 2023-10-21T03:52:10.598 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-21T03:52:10.604 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-21T03:52:10.604 INFO:teuthology.run_tasks:Running task console_log... 2023-10-21T03:52:10.644 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4185cfaa60>, signals=[15]) 2023-10-21T03:52:10.646 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-21T03:52:10.652 INFO:teuthology.task.internal:Opening connections... 2023-10-21T03:52:10.652 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2023-10-21T03:52:10.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:52:10.730 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-21T03:52:10.736 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2023-10-21T03:52:10.843 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2023-10-21T03:52:10.843 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2023-10-21T03:52:10.891 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-21T03:52:10.891 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="Ubuntu" 2023-10-21T03:52:10.892 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="22.04" 2023-10-21T03:52:10.892 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-21T03:52:10.892 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_CODENAME=jammy 2023-10-21T03:52:10.892 INFO:teuthology.orchestra.run.smithi119.stdout:ID=ubuntu 2023-10-21T03:52:10.892 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE=debian 2023-10-21T03:52:10.893 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-21T03:52:10.893 INFO:teuthology.orchestra.run.smithi119.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-21T03:52:10.893 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-21T03:52:10.893 INFO:teuthology.orchestra.run.smithi119.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-21T03:52:10.893 INFO:teuthology.orchestra.run.smithi119.stdout:UBUNTU_CODENAME=jammy 2023-10-21T03:52:10.894 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2023-10-21T03:52:10.922 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-21T03:52:10.931 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-21T03:52:10.937 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-21T03:52:10.938 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2023-10-21T03:52:10.943 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-21T03:52:10.948 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-21T03:52:10.949 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2023-10-21T03:52:11.016 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-21T03:52:11.049 INFO:teuthology.run_tasks:Running task kernel... 2023-10-21T03:52:11.063 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-21T03:52:11.066 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-21T03:52:11.066 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-21T03:52:11.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T03:52:11.071 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2023-10-21T03:52:11.119 INFO:teuthology.orchestra.run.smithi119.stdout:5.15.0-56-generic 2023-10-21T03:52:11.119 INFO:teuthology.task.kernel:Running kernel on smithi119: 5.15.0-56-generic 2023-10-21T03:52:11.119 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get clean 2023-10-21T03:52:11.234 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get update 2023-10-21T03:52:11.509 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-21T03:52:11.583 INFO:teuthology.orchestra.run.smithi119.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-21T03:52:11.584 INFO:teuthology.orchestra.run.smithi119.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-21T03:52:11.926 INFO:teuthology.orchestra.run.smithi119.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-21T03:52:12.015 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [353 kB] 2023-10-21T03:52:12.090 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,103 kB] 2023-10-21T03:52:12.285 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [896 kB] 2023-10-21T03:52:12.431 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [180 kB] 2023-10-21T03:52:12.445 INFO:teuthology.orchestra.run.smithi119.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [515 kB] 2023-10-21T03:52:12.451 INFO:teuthology.orchestra.run.smithi119.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-21T03:52:12.452 INFO:teuthology.orchestra.run.smithi119.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-21T03:52:12.454 INFO:teuthology.orchestra.run.smithi119.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,016 kB] 2023-10-21T03:52:12.472 INFO:teuthology.orchestra.run.smithi119.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [239 kB] 2023-10-21T03:52:12.501 INFO:teuthology.orchestra.run.smithi119.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-21T03:52:12.501 INFO:teuthology.orchestra.run.smithi119.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,036 kB] 2023-10-21T03:52:12.519 INFO:teuthology.orchestra.run.smithi119.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [164 kB] 2023-10-21T03:52:12.529 INFO:teuthology.orchestra.run.smithi119.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-10-21T03:52:12.529 INFO:teuthology.orchestra.run.smithi119.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [793 kB] 2023-10-21T03:52:12.558 INFO:teuthology.orchestra.run.smithi119.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-10-21T03:52:12.560 INFO:teuthology.orchestra.run.smithi119.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [167 kB] 2023-10-21T03:52:12.562 INFO:teuthology.orchestra.run.smithi119.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-10-21T03:52:12.568 INFO:teuthology.orchestra.run.smithi119.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-10-21T03:52:12.732 INFO:teuthology.orchestra.run.smithi119.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-10-21T03:52:12.742 INFO:teuthology.orchestra.run.smithi119.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-21T03:52:13.126 INFO:teuthology.orchestra.run.smithi119.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-10-21T03:52:13.133 INFO:teuthology.orchestra.run.smithi119.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-21T03:52:13.148 INFO:teuthology.orchestra.run.smithi119.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-21T03:52:13.149 INFO:teuthology.orchestra.run.smithi119.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-21T03:52:13.152 INFO:teuthology.orchestra.run.smithi119.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-21T03:52:13.302 INFO:teuthology.orchestra.run.smithi119.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [995 kB] 2023-10-21T03:52:13.412 INFO:teuthology.orchestra.run.smithi119.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-10-21T03:52:13.431 INFO:teuthology.orchestra.run.smithi119.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-21T03:52:13.433 INFO:teuthology.orchestra.run.smithi119.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-21T03:52:13.433 INFO:teuthology.orchestra.run.smithi119.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-21T03:52:13.436 INFO:teuthology.orchestra.run.smithi119.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-21T03:52:13.437 INFO:teuthology.orchestra.run.smithi119.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-21T03:52:13.437 INFO:teuthology.orchestra.run.smithi119.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-10-21T03:52:13.442 INFO:teuthology.orchestra.run.smithi119.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-10-21T03:52:13.446 INFO:teuthology.orchestra.run.smithi119.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-21T03:52:13.610 INFO:teuthology.orchestra.run.smithi119.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-21T03:52:13.692 INFO:teuthology.orchestra.run.smithi119.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-10-21T03:52:13.774 INFO:teuthology.orchestra.run.smithi119.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-10-21T03:52:13.828 INFO:teuthology.orchestra.run.smithi119.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-21T03:52:13.885 INFO:teuthology.orchestra.run.smithi119.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-10-21T03:52:18.241 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 9,805 kB in 3s (3,808 kB/s) 2023-10-21T03:52:19.618 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2023-10-21T03:52:19.636 DEBUG:teuthology.orchestra.run.smithi119:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-21T03:52:19.698 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2023-10-21T03:52:19.884 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2023-10-21T03:52:19.885 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2023-10-21T03:52:20.042 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T03:52:20.042 INFO:teuthology.orchestra.run.smithi119.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-21T03:52:20.043 INFO:teuthology.orchestra.run.smithi119.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-21T03:52:20.043 INFO:teuthology.orchestra.run.smithi119.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-21T03:52:20.043 INFO:teuthology.orchestra.run.smithi119.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-21T03:52:20.044 INFO:teuthology.orchestra.run.smithi119.stdout: python2.7-minimal 2023-10-21T03:52:20.044 INFO:teuthology.orchestra.run.smithi119.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T03:52:20.044 INFO:teuthology.orchestra.run.smithi119.stdout:The following additional packages will be installed: 2023-10-21T03:52:20.045 INFO:teuthology.orchestra.run.smithi119.stdout: linux-generic linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-21T03:52:20.045 INFO:teuthology.orchestra.run.smithi119.stdout: linux-headers-generic linux-image-5.15.0-87-generic 2023-10-21T03:52:20.045 INFO:teuthology.orchestra.run.smithi119.stdout: linux-modules-5.15.0-87-generic linux-modules-extra-5.15.0-87-generic 2023-10-21T03:52:20.046 INFO:teuthology.orchestra.run.smithi119.stdout:Suggested packages: 2023-10-21T03:52:20.046 INFO:teuthology.orchestra.run.smithi119.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-21T03:52:20.046 INFO:teuthology.orchestra.run.smithi119.stdout:Recommended packages: 2023-10-21T03:52:20.046 INFO:teuthology.orchestra.run.smithi119.stdout: thermald 2023-10-21T03:52:20.086 INFO:teuthology.orchestra.run.smithi119.stdout:The following NEW packages will be installed: 2023-10-21T03:52:20.087 INFO:teuthology.orchestra.run.smithi119.stdout: linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-21T03:52:20.087 INFO:teuthology.orchestra.run.smithi119.stdout: linux-image-5.15.0-87-generic linux-modules-5.15.0-87-generic 2023-10-21T03:52:20.087 INFO:teuthology.orchestra.run.smithi119.stdout: linux-modules-extra-5.15.0-87-generic 2023-10-21T03:52:20.088 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages will be upgraded: 2023-10-21T03:52:20.088 INFO:teuthology.orchestra.run.smithi119.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-21T03:52:20.137 INFO:teuthology.orchestra.run.smithi119.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-10-21T03:52:20.138 INFO:teuthology.orchestra.run.smithi119.stdout:Need to get 113 MB of archives. 2023-10-21T03:52:20.138 INFO:teuthology.orchestra.run.smithi119.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-21T03:52:20.138 INFO:teuthology.orchestra.run.smithi119.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-87-generic amd64 5.15.0-87.97 [22.4 MB] 2023-10-21T03:52:20.555 INFO:teuthology.orchestra.run.smithi119.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-87-generic amd64 5.15.0-87.97 [11.5 MB] 2023-10-21T03:52:20.705 INFO:teuthology.orchestra.run.smithi119.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-87-generic amd64 5.15.0-87.97 [64.0 MB] 2023-10-21T03:52:21.575 INFO:teuthology.orchestra.run.smithi119.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.87.84 [1,698 B] 2023-10-21T03:52:21.576 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.87.84 [2,482 B] 2023-10-21T03:52:21.576 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87 all 5.15.0-87.97 [12.3 MB] 2023-10-21T03:52:21.706 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87-generic amd64 5.15.0-87.97 [2,888 kB] 2023-10-21T03:52:21.799 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.87.84 [2,322 B] 2023-10-21T03:52:22.182 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 113 MB in 2s (66.3 MB/s) 2023-10-21T03:52:22.356 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-modules-5.15.0-87-generic. 2023-10-21T03:52:24.354 INFO:teuthology.orchestra.run.smithi119.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-21T03:52:24.358 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../0-linux-modules-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-21T03:52:24.453 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:26.631 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-image-5.15.0-87-generic. 2023-10-21T03:52:26.652 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../1-linux-image-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-21T03:52:26.731 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:27.201 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-87-generic. 2023-10-21T03:52:27.221 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-21T03:52:27.259 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:34.325 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../3-linux-generic_5.15.0.87.84_amd64.deb ... 2023-10-21T03:52:34.452 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-21T03:52:34.910 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.87.84_amd64.deb ... 2023-10-21T03:52:35.022 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-image-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-21T03:52:35.308 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-headers-5.15.0-87. 2023-10-21T03:52:35.331 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../5-linux-headers-5.15.0-87_5.15.0-87.97_all.deb ... 2023-10-21T03:52:35.374 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-21T03:52:41.414 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-headers-5.15.0-87-generic. 2023-10-21T03:52:41.450 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../6-linux-headers-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-21T03:52:41.480 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:44.129 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.87.84_amd64.deb ... 2023-10-21T03:52:44.258 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-headers-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-21T03:52:44.582 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-21T03:52:44.721 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:44.847 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-headers-generic (5.15.0.87.84) ... 2023-10-21T03:52:44.974 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:46.394 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:47.801 INFO:teuthology.orchestra.run.smithi119.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-21T03:52:47.801 INFO:teuthology.orchestra.run.smithi119.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-21T03:52:47.801 INFO:teuthology.orchestra.run.smithi119.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-87-generic 2023-10-21T03:52:47.801 INFO:teuthology.orchestra.run.smithi119.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-87-generic 2023-10-21T03:52:47.954 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:49.330 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-image-generic (5.15.0.87.84) ... 2023-10-21T03:52:49.456 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-generic (5.15.0.87.84) ... 2023-10-21T03:52:49.599 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:52:49.660 INFO:teuthology.orchestra.run.smithi119.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-21T03:52:49.660 INFO:teuthology.orchestra.run.smithi119.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-87-generic 2023-10-21T03:53:03.934 INFO:teuthology.orchestra.run.smithi119.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-21T03:53:03.934 INFO:teuthology.orchestra.run.smithi119.stdout:Sourcing file `/etc/default/grub' 2023-10-21T03:53:03.953 INFO:teuthology.orchestra.run.smithi119.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-21T03:53:03.972 INFO:teuthology.orchestra.run.smithi119.stdout:Generating grub configuration file ... 2023-10-21T03:53:04.270 INFO:teuthology.orchestra.run.smithi119.stdout:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-21T03:53:04.293 INFO:teuthology.orchestra.run.smithi119.stdout:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-21T03:53:04.635 INFO:teuthology.orchestra.run.smithi119.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-21T03:53:04.641 INFO:teuthology.orchestra.run.smithi119.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-21T03:53:04.711 INFO:teuthology.orchestra.run.smithi119.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-21T03:53:04.716 INFO:teuthology.orchestra.run.smithi119.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-21T03:53:04.892 INFO:teuthology.orchestra.run.smithi119.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-21T03:53:04.892 INFO:teuthology.orchestra.run.smithi119.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-21T03:53:04.892 INFO:teuthology.orchestra.run.smithi119.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-21T03:53:04.934 INFO:teuthology.orchestra.run.smithi119.stdout:done 2023-10-21T03:53:05.402 DEBUG:teuthology.orchestra.run.smithi119:> dpkg -s linux-image-generic 2023-10-21T03:53:05.426 INFO:teuthology.orchestra.run.smithi119.stdout:Package: linux-image-generic 2023-10-21T03:53:05.427 INFO:teuthology.orchestra.run.smithi119.stdout:Status: install ok installed 2023-10-21T03:53:05.427 INFO:teuthology.orchestra.run.smithi119.stdout:Priority: optional 2023-10-21T03:53:05.427 INFO:teuthology.orchestra.run.smithi119.stdout:Section: kernel 2023-10-21T03:53:05.427 INFO:teuthology.orchestra.run.smithi119.stdout:Installed-Size: 20 2023-10-21T03:53:05.427 INFO:teuthology.orchestra.run.smithi119.stdout:Maintainer: Ubuntu Kernel Team 2023-10-21T03:53:05.428 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: amd64 2023-10-21T03:53:05.428 INFO:teuthology.orchestra.run.smithi119.stdout:Source: linux-meta 2023-10-21T03:53:05.428 INFO:teuthology.orchestra.run.smithi119.stdout:Version: 5.15.0.87.84 2023-10-21T03:53:05.428 INFO:teuthology.orchestra.run.smithi119.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-21T03:53:05.428 INFO:teuthology.orchestra.run.smithi119.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-21T03:53:05.429 INFO:teuthology.orchestra.run.smithi119.stdout:Recommends: thermald 2023-10-21T03:53:05.429 INFO:teuthology.orchestra.run.smithi119.stdout:Description: Generic Linux kernel image 2023-10-21T03:53:05.429 INFO:teuthology.orchestra.run.smithi119.stdout: This package will always depend on the latest generic kernel image 2023-10-21T03:53:05.429 INFO:teuthology.orchestra.run.smithi119.stdout: available. 2023-10-21T03:53:05.429 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-87-generic 2023-10-21T03:53:05.430 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-21T03:53:05.430 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-21T03:53:05.430 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-21T03:53:05.430 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2023-10-21T03:53:05.431 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get clean 2023-10-21T03:53:05.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get update 2023-10-21T03:53:05.717 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-21T03:53:05.818 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-21T03:53:05.897 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-21T03:53:05.978 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-21T03:53:07.042 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2023-10-21T03:53:07.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-21T03:53:07.121 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2023-10-21T03:53:07.307 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2023-10-21T03:53:07.307 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2023-10-21T03:53:07.468 INFO:teuthology.orchestra.run.smithi119.stdout:linux-image-generic is already the newest version (5.15.0.87.84). 2023-10-21T03:53:07.469 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T03:53:07.469 INFO:teuthology.orchestra.run.smithi119.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-21T03:53:07.469 INFO:teuthology.orchestra.run.smithi119.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-21T03:53:07.469 INFO:teuthology.orchestra.run.smithi119.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-21T03:53:07.470 INFO:teuthology.orchestra.run.smithi119.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-21T03:53:07.470 INFO:teuthology.orchestra.run.smithi119.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-21T03:53:07.470 INFO:teuthology.orchestra.run.smithi119.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T03:53:07.509 INFO:teuthology.orchestra.run.smithi119.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-10-21T03:53:07.511 DEBUG:teuthology.orchestra.run.smithi119:> dpkg -s linux-image-generic 2023-10-21T03:53:07.525 INFO:teuthology.orchestra.run.smithi119.stdout:Package: linux-image-generic 2023-10-21T03:53:07.525 INFO:teuthology.orchestra.run.smithi119.stdout:Status: install ok installed 2023-10-21T03:53:07.525 INFO:teuthology.orchestra.run.smithi119.stdout:Priority: optional 2023-10-21T03:53:07.525 INFO:teuthology.orchestra.run.smithi119.stdout:Section: kernel 2023-10-21T03:53:07.526 INFO:teuthology.orchestra.run.smithi119.stdout:Installed-Size: 20 2023-10-21T03:53:07.526 INFO:teuthology.orchestra.run.smithi119.stdout:Maintainer: Ubuntu Kernel Team 2023-10-21T03:53:07.526 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: amd64 2023-10-21T03:53:07.526 INFO:teuthology.orchestra.run.smithi119.stdout:Source: linux-meta 2023-10-21T03:53:07.527 INFO:teuthology.orchestra.run.smithi119.stdout:Version: 5.15.0.87.84 2023-10-21T03:53:07.527 INFO:teuthology.orchestra.run.smithi119.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-21T03:53:07.527 INFO:teuthology.orchestra.run.smithi119.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-21T03:53:07.527 INFO:teuthology.orchestra.run.smithi119.stdout:Recommends: thermald 2023-10-21T03:53:07.527 INFO:teuthology.orchestra.run.smithi119.stdout:Description: Generic Linux kernel image 2023-10-21T03:53:07.528 INFO:teuthology.orchestra.run.smithi119.stdout: This package will always depend on the latest generic kernel image 2023-10-21T03:53:07.528 INFO:teuthology.orchestra.run.smithi119.stdout: available. 2023-10-21T03:53:07.528 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2023-10-21T03:53:07.570 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.OTP6wfeHMG 2023-10-21T03:53:07.570 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub/grub.cfg /tmp/tmp.OTP6wfeHMG 2023-10-21T03:53:07.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.OTP6wfeHMG 2023-10-21T03:53:07.754 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.OTP6wfeHMG is 10KB 2023-10-21T03:53:07.769 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.OTP6wfeHMG 2023-10-21T03:53:07.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-21T03:53:07.829 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T03:53:07.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-21T03:53:07.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-21T03:53:07.897 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-87-generic 2023-10-21T03:53:07.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo update-grub 2023-10-21T03:53:09.182 INFO:teuthology.orchestra.run.smithi119.stderr:Sourcing file `/etc/default/grub' 2023-10-21T03:53:09.182 INFO:teuthology.orchestra.run.smithi119.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-21T03:53:09.184 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2023-10-21T03:53:09.355 INFO:teuthology.orchestra.run.smithi119.stderr:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-21T03:53:09.360 INFO:teuthology.orchestra.run.smithi119.stderr:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-21T03:53:09.616 INFO:teuthology.orchestra.run.smithi119.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-21T03:53:09.621 INFO:teuthology.orchestra.run.smithi119.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-21T03:53:09.664 INFO:teuthology.orchestra.run.smithi119.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-21T03:53:09.669 INFO:teuthology.orchestra.run.smithi119.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-21T03:53:09.788 INFO:teuthology.orchestra.run.smithi119.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-21T03:53:09.789 INFO:teuthology.orchestra.run.smithi119.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-21T03:53:09.789 INFO:teuthology.orchestra.run.smithi119.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-21T03:53:09.807 INFO:teuthology.orchestra.run.smithi119.stderr:done 2023-10-21T03:53:09.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2023-10-21T03:53:39.846 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-21T03:53:39.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2023-10-21T03:53:39.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:53:58.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2023-10-21T03:54:07.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2023-10-21T03:54:07.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:54:10.458 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2023-10-21T03:54:22.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2023-10-21T03:54:22.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:54:40.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2023-10-21T03:54:55.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2023-10-21T03:54:55.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:54:56.572 DEBUG:teuthology.orchestra.run.smithi119:> true 2023-10-21T03:54:57.499 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2023-10-21T03:54:57.500 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-87-generic"... 2023-10-21T03:54:57.500 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2023-10-21T03:54:57.546 INFO:teuthology.orchestra.run.smithi119.stdout:5.15.0-87-generic 2023-10-21T03:54:57.546 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-87-generic vs 5.15.0-87-generic 2023-10-21T03:54:57.547 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-21T03:54:57.547 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-21T03:54:58.548 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-21T03:54:58.549 DEBUG:teuthology.orchestra.run.smithi119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-21T03:54:58.638 INFO:teuthology.orchestra.run.smithi119.stdout:ttyS1 2023-10-21T03:54:58.680 DEBUG:teuthology.parallel:result is None 2023-10-21T03:54:58.681 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-21T03:54:58.688 INFO:teuthology.task.internal:Creating test directory... 2023-10-21T03:54:58.688 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-21T03:54:58.694 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-21T03:54:58.702 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-21T03:54:58.708 INFO:teuthology.task.internal:Creating archive directory... 2023-10-21T03:54:58.709 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-21T03:54:58.758 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-21T03:54:58.766 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-21T03:54:58.766 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-21T03:54:58.822 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-21T03:54:58.830 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-21T03:54:58.832 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-21T03:54:58.840 INFO:teuthology.task.internal:Configuring sudo... 2023-10-21T03:54:58.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-21T03:54:58.888 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-21T03:54:58.897 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-21T03:54:58.898 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-21T03:54:58.935 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-21T03:54:58.983 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-21T03:54:59.035 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T03:54:59.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-21T03:54:59.100 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2023-10-21T03:54:59.237 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-21T03:54:59.246 INFO:teuthology.task.internal:Starting timer... 2023-10-21T03:54:59.246 INFO:teuthology.run_tasks:Running task pcp... 2023-10-21T03:54:59.258 INFO:teuthology.run_tasks:Running task selinux... 2023-10-21T03:54:59.266 DEBUG:teuthology.task.selinux:Excluding smithi119: OS 'ubuntu' does not support SELinux 2023-10-21T03:54:59.267 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-21T03:54:59.267 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-21T03:54:59.267 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-21T03:54:59.267 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-21T03:54:59.276 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-21T03:54:59.277 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-21T03:54:59.297 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-21T03:54:59.299 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi119.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-21T04:02:44.035 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi119.front.sepia.ceph.com')] 2023-10-21T04:02:44.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2023-10-21T04:02:44.038 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T04:02:44.115 DEBUG:teuthology.orchestra.run.smithi119:> true 2023-10-21T04:02:44.226 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2023-10-21T04:02:44.226 INFO:teuthology.run_tasks:Running task clock... 2023-10-21T04:02:44.238 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-21T04:02:44.239 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-21T04:02:44.240 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-21T04:02:44.300 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-21T04:02:44.300 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: Command line: ntpd -gq 2023-10-21T04:02:44.301 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: ---------------------------------------------------- 2023-10-21T04:02:44.301 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: ntp-4 is maintained by Network Time Foundation, 2023-10-21T04:02:44.301 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-21T04:02:44.302 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: corporation. Support and training for ntp-4 are 2023-10-21T04:02:44.302 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: available at https://www.nwtime.org/support 2023-10-21T04:02:44.302 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: ---------------------------------------------------- 2023-10-21T04:02:44.303 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: proto: precision = 0.063 usec (-24) 2023-10-21T04:02:44.303 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: basedate set to 2022-02-04 2023-10-21T04:02:44.304 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: gps base set to 2022-02-06 (week 2196) 2023-10-21T04:02:44.304 INFO:teuthology.orchestra.run.smithi119.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-21T04:02:44.305 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-21T04:02:44.306 INFO:teuthology.orchestra.run.smithi119.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-21T04:02:44.307 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: restrict ::: KOD does nothing without LIMITED. 2023-10-21T04:02:44.308 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: Listen and drop on 0 v6wildcard [::]:123 2023-10-21T04:02:44.308 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-21T04:02:44.308 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: Listen normally on 2 lo 127.0.0.1:123 2023-10-21T04:02:44.309 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: Listen normally on 3 enp3s0f1 172.21.15.119:123 2023-10-21T04:02:44.309 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: Listen normally on 4 lo [::1]:123 2023-10-21T04:02:44.310 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d5ef%5]:123 2023-10-21T04:02:44.310 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:44 ntpd[28068]: Listening on routing socket on fd #22 for interface updates 2023-10-21T04:02:45.297 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:45 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:45.297 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:45 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:45.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:45 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:45.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:45 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:45.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:45 ntpd[28068]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:02:45.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:45 ntpd[28068]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:02:46.297 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:46 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:46.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:46 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:46.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:46 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:46.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:46 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:46.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:46 ntpd[28068]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:02:46.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:46 ntpd[28068]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:02:47.297 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:47 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:47.297 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:47 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:47.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:47 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:47.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:47 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:47.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:47 ntpd[28068]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:02:47.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:47 ntpd[28068]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:02:48.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:48 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:48.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:48 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:48.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:48 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:48.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:48 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:48.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:48 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:48.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:48 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:48.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:48 ntpd[28068]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:02:48.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:48 ntpd[28068]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:02:48.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:48 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:48.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:48 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:48.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:48 ntpd[28068]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:02:48.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:48 ntpd[28068]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:02:49.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:49 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:49.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:49 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:49.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:49 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:49.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:49 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:49.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:49 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:49.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:49 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:49.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:49 ntpd[28068]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:02:49.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:49 ntpd[28068]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:02:50.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:50 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:50.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:50 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:50.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:50 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:50.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:50 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:50.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:50 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:50.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:50 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:50.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:50 ntpd[28068]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:02:50.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:50 ntpd[28068]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:02:50.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:50 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:50.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:50 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:50.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:50 ntpd[28068]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:02:50.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:50 ntpd[28068]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:02:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:51 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:51 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:51.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:51 ntpd[28068]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T04:02:51.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:51 ntpd[28068]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T04:02:51.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:51 ntpd[28068]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T04:02:51.298 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:51 ntpd[28068]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T04:02:51.299 INFO:teuthology.orchestra.run.smithi119.stdout:21 Oct 04:02:51 ntpd[28068]: ntpd: time slew -0.001425 s 2023-10-21T04:02:51.299 INFO:teuthology.orchestra.run.smithi119.stdout:ntpd: time slew -0.001425s 2023-10-21T04:02:51.299 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:51 ntpd[28068]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-21T04:02:51.300 INFO:teuthology.orchestra.run.smithi119.stderr:21 Oct 04:02:51 ntpd[28068]: can't open /var/log/ntpstats/loopstats.20231021: Permission denied 2023-10-21T04:02:51.400 INFO:teuthology.orchestra.run.smithi119.stdout: remote refid st t when poll reach delay offset jitter 2023-10-21T04:02:51.400 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================== 2023-10-21T04:02:51.400 INFO:teuthology.orchestra.run.smithi119.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-21T04:02:51.401 INFO:teuthology.orchestra.run.smithi119.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-21T04:02:51.401 INFO:teuthology.orchestra.run.smithi119.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-21T04:02:51.401 INFO:teuthology.orchestra.run.smithi119.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-21T04:02:51.405 INFO:teuthology.run_tasks:Running task install... 2023-10-21T04:02:51.415 DEBUG:teuthology.task.install:project ceph 2023-10-21T04:02:51.415 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}} 2023-10-21T04:02:51.415 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'} 2023-10-21T04:02:51.416 INFO:teuthology.task.install:Using flavor: default 2023-10-21T04:02:51.422 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-21T04:02:51.423 INFO:teuthology.task.install:extra packages: [] 2023-10-21T04:02:51.423 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-key list | grep Ceph 2023-10-21T04:02:51.506 INFO:teuthology.orchestra.run.smithi119.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-21T04:02:51.535 INFO:teuthology.orchestra.run.smithi119.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-21T04:02:51.535 INFO:teuthology.orchestra.run.smithi119.stdout:uid [ unknown] Ceph.com (release key) 2023-10-21T04:02:51.536 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-21T04:02:51.536 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:02:51.783 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default/ 2023-10-21T04:02:51.784 INFO:teuthology.task.install.deb:Package version is 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:02:51.979 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:02:51.980 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-21T04:02:51.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get update 2023-10-21T04:02:52.159 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-21T04:02:52.214 INFO:teuthology.orchestra.run.smithi119.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy InRelease 2023-10-21T04:02:52.236 INFO:teuthology.orchestra.run.smithi119.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2023-10-21T04:02:52.259 INFO:teuthology.orchestra.run.smithi119.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-21T04:02:52.279 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-21T04:02:52.281 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2023-10-21T04:02:52.309 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-21T04:02:52.361 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-21T04:02:52.443 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-21T04:02:52.580 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 27.9 kB in 0s (68.0 kB/s) 2023-10-21T04:02:53.416 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2023-10-21T04:02:53.431 DEBUG:teuthology.orchestra.run.smithi119:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6744-g7f103c7c-1jammy cephadm=18.0.0-6744-g7f103c7c-1jammy ceph-mds=18.0.0-6744-g7f103c7c-1jammy ceph-mgr=18.0.0-6744-g7f103c7c-1jammy ceph-common=18.0.0-6744-g7f103c7c-1jammy ceph-fuse=18.0.0-6744-g7f103c7c-1jammy ceph-test=18.0.0-6744-g7f103c7c-1jammy ceph-volume=18.0.0-6744-g7f103c7c-1jammy radosgw=18.0.0-6744-g7f103c7c-1jammy python3-rados=18.0.0-6744-g7f103c7c-1jammy python3-rgw=18.0.0-6744-g7f103c7c-1jammy python3-cephfs=18.0.0-6744-g7f103c7c-1jammy python3-rbd=18.0.0-6744-g7f103c7c-1jammy libcephfs2=18.0.0-6744-g7f103c7c-1jammy libcephfs-dev=18.0.0-6744-g7f103c7c-1jammy librados2=18.0.0-6744-g7f103c7c-1jammy librbd1=18.0.0-6744-g7f103c7c-1jammy rbd-fuse=18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:02:53.482 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2023-10-21T04:02:53.646 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2023-10-21T04:02:53.647 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2023-10-21T04:02:53.794 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T04:02:53.794 INFO:teuthology.orchestra.run.smithi119.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-21T04:02:53.795 INFO:teuthology.orchestra.run.smithi119.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-21T04:02:53.795 INFO:teuthology.orchestra.run.smithi119.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-21T04:02:53.795 INFO:teuthology.orchestra.run.smithi119.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-21T04:02:53.795 INFO:teuthology.orchestra.run.smithi119.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-21T04:02:53.795 INFO:teuthology.orchestra.run.smithi119.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T04:02:53.796 INFO:teuthology.orchestra.run.smithi119.stdout:The following additional packages will be installed: 2023-10-21T04:02:53.797 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-21T04:02:53.797 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-21T04:02:53.797 INFO:teuthology.orchestra.run.smithi119.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-21T04:02:53.797 INFO:teuthology.orchestra.run.smithi119.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-21T04:02:53.797 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-21T04:02:53.798 INFO:teuthology.orchestra.run.smithi119.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-21T04:02:53.798 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-21T04:02:53.798 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-21T04:02:53.798 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-21T04:02:53.798 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-21T04:02:53.799 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-21T04:02:53.799 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-21T04:02:53.799 INFO:teuthology.orchestra.run.smithi119.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-21T04:02:53.799 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-21T04:02:53.799 INFO:teuthology.orchestra.run.smithi119.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-21T04:02:53.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-21T04:02:53.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-21T04:02:53.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-21T04:02:53.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-21T04:02:53.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-21T04:02:53.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-21T04:02:53.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-21T04:02:53.801 INFO:teuthology.orchestra.run.smithi119.stdout:Suggested packages: 2023-10-21T04:02:53.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-21T04:02:53.802 INFO:teuthology.orchestra.run.smithi119.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-21T04:02:53.802 INFO:teuthology.orchestra.run.smithi119.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-21T04:02:53.802 INFO:teuthology.orchestra.run.smithi119.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-21T04:02:53.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-21T04:02:53.803 INFO:teuthology.orchestra.run.smithi119.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-21T04:02:53.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-watchdog 2023-10-21T04:02:53.803 INFO:teuthology.orchestra.run.smithi119.stdout:Recommended packages: 2023-10-21T04:02:53.803 INFO:teuthology.orchestra.run.smithi119.stdout: btrfs-tools 2023-10-21T04:02:53.890 INFO:teuthology.orchestra.run.smithi119.stdout:The following NEW packages will be installed: 2023-10-21T04:02:53.891 INFO:teuthology.orchestra.run.smithi119.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-21T04:02:53.891 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-21T04:02:53.891 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-21T04:02:53.891 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-21T04:02:53.892 INFO:teuthology.orchestra.run.smithi119.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-21T04:02:53.892 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-21T04:02:53.893 INFO:teuthology.orchestra.run.smithi119.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-21T04:02:53.893 INFO:teuthology.orchestra.run.smithi119.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-21T04:02:53.893 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-21T04:02:53.894 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-21T04:02:53.894 INFO:teuthology.orchestra.run.smithi119.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-21T04:02:53.894 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-21T04:02:53.894 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-21T04:02:53.895 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-21T04:02:53.895 INFO:teuthology.orchestra.run.smithi119.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-21T04:02:53.895 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-21T04:02:53.895 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-21T04:02:53.895 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-21T04:02:53.896 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-21T04:02:53.896 INFO:teuthology.orchestra.run.smithi119.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-21T04:02:53.896 INFO:teuthology.orchestra.run.smithi119.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-21T04:02:53.896 INFO:teuthology.orchestra.run.smithi119.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-21T04:02:53.896 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-21T04:02:53.896 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-21T04:02:53.897 INFO:teuthology.orchestra.run.smithi119.stdout: socat xmlstarlet 2023-10-21T04:02:54.036 INFO:teuthology.orchestra.run.smithi119.stdout:0 upgraded, 109 newly installed, 0 to remove and 257 not upgraded. 2023-10-21T04:02:54.036 INFO:teuthology.orchestra.run.smithi119.stdout:Need to get 184 MB of archives. 2023-10-21T04:02:54.036 INFO:teuthology.orchestra.run.smithi119.stdout:After this operation, 824 MB of additional disk space will be used. 2023-10-21T04:02:54.036 INFO:teuthology.orchestra.run.smithi119.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6744-g7f103c7c-1jammy [3,450 kB] 2023-10-21T04:02:54.078 INFO:teuthology.orchestra.run.smithi119.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-21T04:02:54.324 INFO:teuthology.orchestra.run.smithi119.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6744-g7f103c7c-1jammy [3,132 kB] 2023-10-21T04:02:54.357 INFO:teuthology.orchestra.run.smithi119.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-21T04:02:54.442 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6744-g7f103c7c-1jammy [689 kB] 2023-10-21T04:02:54.473 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6744-g7f103c7c-1jammy [356 kB] 2023-10-21T04:02:54.483 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6744-g7f103c7c-1jammy [32.8 kB] 2023-10-21T04:02:54.484 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6744-g7f103c7c-1jammy [180 kB] 2023-10-21T04:02:54.488 INFO:teuthology.orchestra.run.smithi119.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6744-g7f103c7c-1jammy [62.4 kB] 2023-10-21T04:02:54.489 INFO:teuthology.orchestra.run.smithi119.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6744-g7f103c7c-1jammy [330 kB] 2023-10-21T04:02:54.503 INFO:teuthology.orchestra.run.smithi119.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6744-g7f103c7c-1jammy [6,221 kB] 2023-10-21T04:02:54.531 INFO:teuthology.orchestra.run.smithi119.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-21T04:02:54.546 INFO:teuthology.orchestra.run.smithi119.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-21T04:02:54.617 INFO:teuthology.orchestra.run.smithi119.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-21T04:02:54.702 INFO:teuthology.orchestra.run.smithi119.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6744-g7f103c7c-1jammy [112 kB] 2023-10-21T04:02:54.705 INFO:teuthology.orchestra.run.smithi119.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6744-g7f103c7c-1jammy [441 kB] 2023-10-21T04:02:54.720 INFO:teuthology.orchestra.run.smithi119.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6744-g7f103c7c-1jammy [22.6 MB] 2023-10-21T04:02:54.814 INFO:teuthology.orchestra.run.smithi119.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-21T04:02:54.823 INFO:teuthology.orchestra.run.smithi119.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-21T04:02:54.854 INFO:teuthology.orchestra.run.smithi119.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-21T04:02:54.863 INFO:teuthology.orchestra.run.smithi119.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-21T04:02:54.864 INFO:teuthology.orchestra.run.smithi119.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-21T04:02:54.866 INFO:teuthology.orchestra.run.smithi119.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-21T04:02:55.057 INFO:teuthology.orchestra.run.smithi119.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-21T04:02:55.279 INFO:teuthology.orchestra.run.smithi119.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-21T04:02:55.293 INFO:teuthology.orchestra.run.smithi119.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-21T04:02:55.419 INFO:teuthology.orchestra.run.smithi119.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-21T04:02:55.427 INFO:teuthology.orchestra.run.smithi119.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-21T04:02:55.438 INFO:teuthology.orchestra.run.smithi119.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-21T04:02:55.454 INFO:teuthology.orchestra.run.smithi119.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-21T04:02:55.527 INFO:teuthology.orchestra.run.smithi119.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-21T04:02:55.528 INFO:teuthology.orchestra.run.smithi119.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-21T04:02:55.529 INFO:teuthology.orchestra.run.smithi119.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-21T04:02:55.580 INFO:teuthology.orchestra.run.smithi119.stdout:Get:34 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6744-g7f103c7c-1jammy [6,852 kB] 2023-10-21T04:02:55.600 INFO:teuthology.orchestra.run.smithi119.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-21T04:02:55.695 INFO:teuthology.orchestra.run.smithi119.stdout:Get:36 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6744-g7f103c7c-1jammy [238 kB] 2023-10-21T04:02:55.701 INFO:teuthology.orchestra.run.smithi119.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6744-g7f103c7c-1jammy [128 kB] 2023-10-21T04:02:55.705 INFO:teuthology.orchestra.run.smithi119.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6744-g7f103c7c-1jammy [1,496 kB] 2023-10-21T04:02:55.756 INFO:teuthology.orchestra.run.smithi119.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6744-g7f103c7c-1jammy [9,863 kB] 2023-10-21T04:02:55.774 INFO:teuthology.orchestra.run.smithi119.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-21T04:02:55.940 INFO:teuthology.orchestra.run.smithi119.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-21T04:02:55.962 INFO:teuthology.orchestra.run.smithi119.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-21T04:02:56.090 INFO:teuthology.orchestra.run.smithi119.stdout:Get:43 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6744-g7f103c7c-1jammy [31.7 MB] 2023-10-21T04:02:56.096 INFO:teuthology.orchestra.run.smithi119.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-21T04:02:56.114 INFO:teuthology.orchestra.run.smithi119.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-21T04:02:56.121 INFO:teuthology.orchestra.run.smithi119.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-21T04:02:56.197 INFO:teuthology.orchestra.run.smithi119.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-21T04:02:56.225 INFO:teuthology.orchestra.run.smithi119.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-21T04:02:56.230 INFO:teuthology.orchestra.run.smithi119.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-21T04:02:56.355 INFO:teuthology.orchestra.run.smithi119.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-21T04:02:56.356 INFO:teuthology.orchestra.run.smithi119.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-21T04:02:56.525 INFO:teuthology.orchestra.run.smithi119.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-21T04:02:56.716 INFO:teuthology.orchestra.run.smithi119.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-21T04:02:56.872 INFO:teuthology.orchestra.run.smithi119.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-21T04:02:56.979 INFO:teuthology.orchestra.run.smithi119.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-21T04:02:57.004 INFO:teuthology.orchestra.run.smithi119.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-21T04:02:57.034 INFO:teuthology.orchestra.run.smithi119.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-21T04:02:57.048 INFO:teuthology.orchestra.run.smithi119.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-21T04:02:57.089 INFO:teuthology.orchestra.run.smithi119.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-21T04:02:57.098 INFO:teuthology.orchestra.run.smithi119.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-21T04:02:57.128 INFO:teuthology.orchestra.run.smithi119.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-21T04:02:57.270 INFO:teuthology.orchestra.run.smithi119.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6744-g7f103c7c-1jammy [14.0 kB] 2023-10-21T04:02:57.270 INFO:teuthology.orchestra.run.smithi119.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6744-g7f103c7c-1jammy [839 kB] 2023-10-21T04:02:57.287 INFO:teuthology.orchestra.run.smithi119.stdout:Get:64 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6744-g7f103c7c-1jammy [2,146 kB] 2023-10-21T04:02:57.326 INFO:teuthology.orchestra.run.smithi119.stdout:Get:65 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6744-g7f103c7c-1jammy [283 kB] 2023-10-21T04:02:57.331 INFO:teuthology.orchestra.run.smithi119.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6744-g7f103c7c-1jammy [138 kB] 2023-10-21T04:02:57.332 INFO:teuthology.orchestra.run.smithi119.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6744-g7f103c7c-1jammy [1,906 kB] 2023-10-21T04:02:57.354 INFO:teuthology.orchestra.run.smithi119.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6744-g7f103c7c-1jammy [8,625 kB] 2023-10-21T04:02:57.354 INFO:teuthology.orchestra.run.smithi119.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-21T04:02:57.522 INFO:teuthology.orchestra.run.smithi119.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-21T04:02:57.614 INFO:teuthology.orchestra.run.smithi119.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6744-g7f103c7c-1jammy [14.2 kB] 2023-10-21T04:02:57.614 INFO:teuthology.orchestra.run.smithi119.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6744-g7f103c7c-1jammy [47.8 MB] 2023-10-21T04:02:57.872 INFO:teuthology.orchestra.run.smithi119.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-21T04:02:57.890 INFO:teuthology.orchestra.run.smithi119.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-21T04:02:57.897 INFO:teuthology.orchestra.run.smithi119.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-21T04:02:57.961 INFO:teuthology.orchestra.run.smithi119.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-21T04:02:58.259 INFO:teuthology.orchestra.run.smithi119.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-21T04:02:58.269 INFO:teuthology.orchestra.run.smithi119.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-21T04:02:58.270 INFO:teuthology.orchestra.run.smithi119.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-21T04:02:58.338 INFO:teuthology.orchestra.run.smithi119.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-21T04:02:58.339 INFO:teuthology.orchestra.run.smithi119.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-21T04:02:58.341 INFO:teuthology.orchestra.run.smithi119.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-21T04:02:58.507 INFO:teuthology.orchestra.run.smithi119.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-21T04:02:58.697 INFO:teuthology.orchestra.run.smithi119.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-21T04:02:58.761 INFO:teuthology.orchestra.run.smithi119.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-21T04:02:58.797 INFO:teuthology.orchestra.run.smithi119.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-21T04:02:58.856 INFO:teuthology.orchestra.run.smithi119.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-21T04:02:59.033 INFO:teuthology.orchestra.run.smithi119.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-21T04:02:59.071 INFO:teuthology.orchestra.run.smithi119.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-21T04:02:59.096 INFO:teuthology.orchestra.run.smithi119.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-21T04:02:59.104 INFO:teuthology.orchestra.run.smithi119.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-21T04:02:59.146 INFO:teuthology.orchestra.run.smithi119.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-21T04:02:59.170 INFO:teuthology.orchestra.run.smithi119.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-21T04:02:59.411 INFO:teuthology.orchestra.run.smithi119.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6744-g7f103c7c-1jammy [120 kB] 2023-10-21T04:02:59.413 INFO:teuthology.orchestra.run.smithi119.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6744-g7f103c7c-1jammy [38.3 kB] 2023-10-21T04:02:59.415 INFO:teuthology.orchestra.run.smithi119.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6744-g7f103c7c-1jammy [12.3 MB] 2023-10-21T04:02:59.422 INFO:teuthology.orchestra.run.smithi119.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-21T04:02:59.705 INFO:teuthology.orchestra.run.smithi119.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6744-g7f103c7c-1jammy [91.6 kB] 2023-10-21T04:02:59.908 INFO:teuthology.orchestra.run.smithi119.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-21T04:02:59.909 INFO:teuthology.orchestra.run.smithi119.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-21T04:02:59.920 INFO:teuthology.orchestra.run.smithi119.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-21T04:02:59.923 INFO:teuthology.orchestra.run.smithi119.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-21T04:02:59.954 INFO:teuthology.orchestra.run.smithi119.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-21T04:02:59.968 INFO:teuthology.orchestra.run.smithi119.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-21T04:03:00.094 INFO:teuthology.orchestra.run.smithi119.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-21T04:03:00.096 INFO:teuthology.orchestra.run.smithi119.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-21T04:03:00.098 INFO:teuthology.orchestra.run.smithi119.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-21T04:03:00.116 INFO:teuthology.orchestra.run.smithi119.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-21T04:03:00.284 INFO:teuthology.orchestra.run.smithi119.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-21T04:03:01.455 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 184 MB in 7s (26.2 MB/s) 2023-10-21T04:03:01.590 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-21T04:03:01.674 INFO:teuthology.orchestra.run.smithi119.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165984 files and directories currently installed.) 2023-10-21T04:03:01.679 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-21T04:03:01.797 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-21T04:03:02.075 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-21T04:03:02.102 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-21T04:03:02.142 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-21T04:03:02.487 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-21T04:03:02.514 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-21T04:03:02.562 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-21T04:03:02.903 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-21T04:03:02.924 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-21T04:03:02.966 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-21T04:03:03.369 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-21T04:03:03.395 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-21T04:03:03.435 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:03:03.813 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-21T04:03:03.839 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-21T04:03:03.879 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:03:04.216 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-21T04:03:04.242 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-21T04:03:04.282 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:03:04.602 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-21T04:03:04.629 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-21T04:03:04.660 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-21T04:03:04.980 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package librados2. 2023-10-21T04:03:05.007 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../008-librados2_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:05.046 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librados2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:05.508 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package librbd1. 2023-10-21T04:03:05.534 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../009-librbd1_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:05.574 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librbd1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:06.044 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libcephfs2. 2023-10-21T04:03:06.070 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:06.110 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libcephfs2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:06.388 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-rados. 2023-10-21T04:03:06.414 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:06.455 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-rados (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:06.716 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-21T04:03:06.742 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:06.774 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-ceph-argparse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:07.026 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cephfs. 2023-10-21T04:03:07.041 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:07.085 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cephfs (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:07.421 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-21T04:03:07.447 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:07.488 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:07.991 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-21T04:03:08.006 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-21T04:03:08.041 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-21T04:03:08.310 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-prettytable. 2023-10-21T04:03:08.336 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-21T04:03:08.368 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-21T04:03:08.613 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-rbd. 2023-10-21T04:03:08.640 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:08.679 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-rbd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:09.032 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-21T04:03:09.059 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-21T04:03:09.099 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-21T04:03:09.393 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package librgw2. 2023-10-21T04:03:09.409 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../019-librgw2_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:09.443 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librgw2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:09.921 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-rgw. 2023-10-21T04:03:09.936 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:09.996 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-rgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:10.332 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-21T04:03:10.358 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-21T04:03:10.398 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-21T04:03:10.693 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libradosstriper1. 2023-10-21T04:03:10.720 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:10.760 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libradosstriper1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:11.188 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-common. 2023-10-21T04:03:11.214 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:11.254 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:12.485 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-base. 2023-10-21T04:03:12.511 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:12.561 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-base (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:13.256 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-21T04:03:13.283 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-21T04:03:13.323 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-21T04:03:13.609 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cheroot. 2023-10-21T04:03:13.635 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-21T04:03:13.675 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-21T04:03:14.004 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-21T04:03:14.030 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-21T04:03:14.095 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-21T04:03:14.356 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-21T04:03:14.383 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-21T04:03:14.415 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-21T04:03:14.701 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-21T04:03:14.727 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-21T04:03:14.767 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-21T04:03:15.081 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-tempora. 2023-10-21T04:03:15.107 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-21T04:03:15.147 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-21T04:03:15.400 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-portend. 2023-10-21T04:03:15.426 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-21T04:03:15.467 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-21T04:03:15.753 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-21T04:03:15.779 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-21T04:03:15.819 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-21T04:03:16.131 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-21T04:03:16.157 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-21T04:03:16.197 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-21T04:03:16.526 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-natsort. 2023-10-21T04:03:16.541 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-21T04:03:16.576 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-21T04:03:16.870 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-logutils. 2023-10-21T04:03:16.897 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-21T04:03:16.937 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-21T04:03:17.249 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-21T04:03:17.275 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-21T04:03:17.315 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-21T04:03:17.618 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-mako. 2023-10-21T04:03:17.644 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-21T04:03:17.684 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-21T04:03:18.013 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-21T04:03:18.039 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-21T04:03:18.080 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-21T04:03:18.591 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-21T04:03:18.617 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-21T04:03:18.657 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-21T04:03:18.927 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-webob. 2023-10-21T04:03:18.953 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-21T04:03:18.985 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-21T04:03:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-waitress. 2023-10-21T04:03:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-21T04:03:19.316 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-21T04:03:19.641 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-tempita. 2023-10-21T04:03:19.667 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-21T04:03:19.707 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-21T04:03:20.010 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-paste. 2023-10-21T04:03:20.037 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-21T04:03:20.077 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-21T04:03:20.388 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-21T04:03:20.414 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-21T04:03:20.455 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-21T04:03:20.758 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-21T04:03:20.784 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-21T04:03:20.824 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-21T04:03:21.136 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-webtest. 2023-10-21T04:03:21.162 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-21T04:03:21.202 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-21T04:03:21.472 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pecan. 2023-10-21T04:03:21.498 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-21T04:03:21.538 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-21T04:03:21.860 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-certifi. 2023-10-21T04:03:21.886 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-21T04:03:21.918 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-21T04:03:22.221 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-idna. 2023-10-21T04:03:22.247 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-21T04:03:22.296 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-21T04:03:22.607 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-urllib3. 2023-10-21T04:03:22.634 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-21T04:03:22.674 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-21T04:03:23.002 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-requests. 2023-10-21T04:03:23.028 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-21T04:03:23.069 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-21T04:03:23.380 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-21T04:03:23.406 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-21T04:03:23.446 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-21T04:03:23.741 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-21T04:03:23.768 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:23.808 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:24.154 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-21T04:03:24.181 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:24.221 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:24.532 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-openssl. 2023-10-21T04:03:24.558 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-21T04:03:24.599 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-21T04:03:24.935 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr. 2023-10-21T04:03:24.961 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:25.002 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:25.380 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mon. 2023-10-21T04:03:25.407 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:25.446 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mon (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:26.433 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-osd. 2023-10-21T04:03:26.460 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:26.500 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-osd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:28.887 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph. 2023-10-21T04:03:28.913 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../059-ceph_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:28.953 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:29.281 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-fuse. 2023-10-21T04:03:29.307 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:29.348 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:29.734 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mds. 2023-10-21T04:03:29.761 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:29.800 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mds (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:30.137 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package cephadm. 2023-10-21T04:03:30.164 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../062-cephadm_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:30.204 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:30.540 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-21T04:03:30.567 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-21T04:03:30.607 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-21T04:03:30.893 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-21T04:03:30.908 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-21T04:03:30.951 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-21T04:03:31.956 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-babel. 2023-10-21T04:03:31.984 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-21T04:03:32.023 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-21T04:03:32.317 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-jinja2. 2023-10-21T04:03:32.344 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-21T04:03:32.384 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-21T04:03:32.687 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-21T04:03:32.713 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:32.754 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:33.082 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-21T04:03:33.109 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-21T04:03:33.149 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-21T04:03:33.452 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-21T04:03:33.478 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-21T04:03:33.535 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-21T04:03:33.930 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-routes. 2023-10-21T04:03:33.956 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-21T04:03:33.996 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-21T04:03:34.277 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-21T04:03:34.301 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:34.341 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:34.778 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-21T04:03:34.793 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-21T04:03:34.836 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-21T04:03:35.332 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-joblib. 2023-10-21T04:03:35.359 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-21T04:03:35.398 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-21T04:03:35.735 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-21T04:03:35.763 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-21T04:03:35.802 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-21T04:03:36.046 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-sklearn. 2023-10-21T04:03:36.073 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-21T04:03:36.113 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-21T04:03:36.796 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-21T04:03:36.813 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:36.846 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:37.833 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cachetools. 2023-10-21T04:03:37.860 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-21T04:03:37.899 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-21T04:03:38.202 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-21T04:03:38.229 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-21T04:03:38.269 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-21T04:03:38.627 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-21T04:03:38.654 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-21T04:03:38.697 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-21T04:03:39.034 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-rsa. 2023-10-21T04:03:39.061 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-21T04:03:39.100 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-21T04:03:39.395 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-google-auth. 2023-10-21T04:03:39.422 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-21T04:03:39.462 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-21T04:03:39.740 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-21T04:03:39.767 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-21T04:03:39.798 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-21T04:03:40.170 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-websocket. 2023-10-21T04:03:40.197 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-21T04:03:40.236 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-21T04:03:40.548 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-21T04:03:40.575 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-21T04:03:40.614 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-21T04:03:41.278 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-21T04:03:41.306 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:41.345 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:41.672 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-21T04:03:41.699 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-21T04:03:41.739 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-21T04:03:42.067 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-21T04:03:42.095 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-21T04:03:42.133 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-21T04:03:42.344 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package jq. 2023-10-21T04:03:42.360 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-21T04:03:42.394 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-21T04:03:42.622 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package socat. 2023-10-21T04:03:42.638 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-21T04:03:42.681 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-21T04:03:43.009 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package xmlstarlet. 2023-10-21T04:03:43.036 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-21T04:03:43.075 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-21T04:03:43.421 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-test. 2023-10-21T04:03:43.437 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:43.479 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-test (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:45.809 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-volume. 2023-10-21T04:03:45.836 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:03:45.875 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-volume (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:46.171 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-21T04:03:46.198 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:46.237 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libcephfs-dev (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:46.566 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package nvme-cli. 2023-10-21T04:03:46.593 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-21T04:03:46.624 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-21T04:03:46.981 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-21T04:03:46.995 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-21T04:03:47.037 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-21T04:03:47.416 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-21T04:03:47.443 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-21T04:03:47.482 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-21T04:03:47.793 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pastescript. 2023-10-21T04:03:47.820 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-21T04:03:47.859 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-21T04:03:48.180 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pluggy. 2023-10-21T04:03:48.207 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-21T04:03:48.246 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-21T04:03:48.566 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-psutil. 2023-10-21T04:03:48.593 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-21T04:03:48.632 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-21T04:03:48.969 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-py. 2023-10-21T04:03:48.996 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-21T04:03:49.043 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-21T04:03:49.381 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pygments. 2023-10-21T04:03:49.408 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-21T04:03:49.455 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-21T04:03:49.885 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-21T04:03:49.912 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-21T04:03:49.942 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-21T04:03:50.228 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-toml. 2023-10-21T04:03:50.256 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-21T04:03:50.295 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-21T04:03:50.598 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pytest. 2023-10-21T04:03:50.625 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-21T04:03:50.664 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-21T04:03:51.018 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-simplejson. 2023-10-21T04:03:51.046 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-21T04:03:51.084 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-21T04:03:51.353 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-21T04:03:51.369 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-21T04:03:51.411 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-21T04:03:52.133 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package radosgw. 2023-10-21T04:03:52.161 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../107-radosgw_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:52.199 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking radosgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:53.010 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package rbd-fuse. 2023-10-21T04:03:53.026 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:03:53.068 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking rbd-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:53.364 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-21T04:03:53.614 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-21T04:03:53.743 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-21T04:03:54.003 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-21T04:03:54.353 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-21T04:03:54.850 INFO:teuthology.orchestra.run.smithi119.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-21T04:03:54.936 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:55.488 INFO:teuthology.orchestra.run.smithi119.stdout:Adding system user cephadm....done 2023-10-21T04:03:55.643 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-21T04:03:55.911 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-21T04:03:56.153 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-21T04:03:56.254 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-21T04:03:56.506 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-21T04:03:56.748 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-21T04:03:56.874 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-21T04:03:57.176 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-21T04:03:57.536 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-21T04:03:57.779 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-21T04:03:58.021 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-21T04:03:58.248 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-21T04:03:58.515 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-21T04:03:58.775 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-ceph-argparse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:03:59.043 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-21T04:03:59.319 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-21T04:03:59.570 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-21T04:03:59.679 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-21T04:04:00.022 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-21T04:04:00.148 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-21T04:04:00.416 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-21T04:04:00.692 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-21T04:04:01.010 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-21T04:04:01.712 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-21T04:04:01.955 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-21T04:04:02.081 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-21T04:04:02.207 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-21T04:04:02.508 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-21T04:04:02.893 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-21T04:04:03.152 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-21T04:04:03.554 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-21T04:04:03.822 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-21T04:04:04.156 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-21T04:04:04.409 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-21T04:04:04.534 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:04.827 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-idna (3.3-1) ... 2023-10-21T04:04:05.136 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-21T04:04:06.234 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:04:06.368 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-21T04:04:06.619 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-21T04:04:06.728 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-21T04:04:06.846 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-21T04:04:07.139 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-21T04:04:07.398 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-21T04:04:07.658 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:04:07.784 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-21T04:04:07.910 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-21T04:04:08.203 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-21T04:04:08.446 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-21T04:04:08.697 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-21T04:04:08.940 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-21T04:04:09.191 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-21T04:04:09.468 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-21T04:04:09.593 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-21T04:04:09.870 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-21T04:04:09.971 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-21T04:04:10.247 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-21T04:04:10.548 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-21T04:04:10.799 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-21T04:04:11.219 INFO:teuthology.orchestra.run.smithi119.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-21T04:04:11.461 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-21T04:04:11.578 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-21T04:04:11.946 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-21T04:04:12.181 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-21T04:04:12.432 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:04:12.558 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-21T04:04:12.901 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-21T04:04:13.202 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-21T04:04:13.470 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-21T04:04:13.596 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-21T04:04:13.872 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-21T04:04:14.131 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-21T04:04:14.383 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-21T04:04:14.801 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-21T04:04:15.094 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-21T04:04:15.438 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-21T04:04:15.580 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librados2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:15.706 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librgw2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:15.833 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:15.959 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libcephfs2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:16.085 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libradosstriper1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:16.288 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-21T04:04:16.556 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librbd1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:16.682 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-modules-core (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:16.808 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:17.029 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-21T04:04:17.030 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-21T04:04:17.551 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libcephfs-dev (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:17.711 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-rados (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:17.837 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-21T04:04:19.000 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-rbd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:19.126 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up rbd-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-rgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:19.353 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cephfs (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:19.454 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:19.964 INFO:teuthology.orchestra.run.smithi119.stdout:Adding group ceph....done 2023-10-21T04:04:20.324 INFO:teuthology.orchestra.run.smithi119.stdout:Adding system user ceph....done 2023-10-21T04:04:20.442 INFO:teuthology.orchestra.run.smithi119.stdout:Setting system user ceph properties....done 2023-10-21T04:04:20.453 INFO:teuthology.orchestra.run.smithi119.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-21T04:04:20.586 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-21T04:04:20.924 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-21T04:04:21.439 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-test (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:21.565 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up radosgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:22.086 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-21T04:04:22.087 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-21T04:04:22.592 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-base (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:22.937 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-21T04:04:23.444 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mds (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:23.680 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-21T04:04:23.680 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-21T04:04:24.171 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:24.358 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-21T04:04:24.359 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-21T04:04:24.864 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-osd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:25.148 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-21T04:04:25.149 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-21T04:04:25.682 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:25.826 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:25.968 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mon (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:26.180 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-21T04:04:26.180 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-21T04:04:26.678 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:26.820 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:26.946 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-dashboard (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:27.097 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-volume (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:04:27.434 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-21T04:04:28.427 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-21T04:04:28.906 INFO:teuthology.orchestra.run.smithi119.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-21T04:04:28.909 DEBUG:teuthology.parallel:result is None 2023-10-21T04:04:28.910 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:04:29.062 DEBUG:teuthology.orchestra.run.smithi119:> dpkg-query -W -f '${Version}' ceph 2023-10-21T04:04:29.081 INFO:teuthology.orchestra.run.smithi119.stdout:18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:04:29.081 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:04:29.082 INFO:teuthology.task.install:The correct ceph version 18.0.0-6744-g7f103c7c-1jammy is installed. 2023-10-21T04:04:29.082 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-21T04:04:29.083 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:29.083 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-21T04:04:29.136 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-21T04:04:29.137 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:29.137 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2023-10-21T04:04:29.191 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-21T04:04:29.246 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-21T04:04:29.247 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:29.247 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-21T04:04:29.298 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-21T04:04:29.350 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-21T04:04:29.351 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:29.351 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2023-10-21T04:04:29.403 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-21T04:04:29.458 INFO:teuthology.run_tasks:Running task ceph... 2023-10-21T04:04:29.524 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-21T04:04:29.525 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /var/log/ceph 2023-10-21T04:04:29.537 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-21T04:04:29.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-21T04:04:29.591 INFO:tasks.ceph:Creating extra log directories... 2023-10-21T04:04:29.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-21T04:04:29.657 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-21T04:04:29.657 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 61}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-21T04:04:29.658 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433375', 'branch': 'ajarr-wip-extra-debug-62994-oct-19', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/defaults msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7433375', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'ajarr-wip-extra-debug-62994-oct-19'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 61}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'ajarr-wip-extra-debug-62994-oct-19', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1449, 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'ajarr-wip-extra-debug-62994-oct-19', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab', 'targets': {'smithi119.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGxiak4qSskTc2wF5c0efzOeCTK+1QFx5H7sKK9+dm0dquM9GdwhHn9zL1QnphPwznKmEs7wAhHBjtwlgK4pd24='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create datapool 4', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '202b180cb047e798fb131047314a862593f45403', 'timestamp': '2023-10-20_15:15:00', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-21T04:04:29.658 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-21T04:04:29.696 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/run/ceph 2023-10-21T04:04:29.753 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:29.754 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2023-10-21T04:04:29.800 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-21T04:04:29.800 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2023-10-21T04:04:29.850 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-21T04:04:29.851 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T04:04:29.851 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 888 Links: 1 2023-10-21T04:04:29.851 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T04:04:29.852 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2023-10-21 03:59:23.702328334 +0000 2023-10-21T04:04:29.852 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2023-10-21 03:59:23.342331549 +0000 2023-10-21T04:04:29.852 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2023-10-21 03:59:23.342331549 +0000 2023-10-21T04:04:29.853 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2023-10-21T04:04:29.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-21T04:04:29.906 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2023-10-21T04:04:29.907 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2023-10-21T04:04:29.907 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000434326 s, 1.2 MB/s 2023-10-21T04:04:29.908 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-21T04:04:29.957 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2023-10-21T04:04:30.004 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-21T04:04:30.004 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T04:04:30.004 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 906 Links: 1 2023-10-21T04:04:30.004 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T04:04:30.005 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2023-10-21 03:59:24.278323188 +0000 2023-10-21T04:04:30.005 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2023-10-21 03:59:23.942326190 +0000 2023-10-21T04:04:30.005 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2023-10-21 03:59:23.942326190 +0000 2023-10-21T04:04:30.005 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2023-10-21T04:04:30.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-21T04:04:30.060 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2023-10-21T04:04:30.060 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2023-10-21T04:04:30.061 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000315481 s, 1.6 MB/s 2023-10-21T04:04:30.062 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-21T04:04:30.109 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2023-10-21T04:04:30.156 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-21T04:04:30.156 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T04:04:30.156 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 916 Links: 1 2023-10-21T04:04:30.156 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T04:04:30.157 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2023-10-21 03:59:24.906317579 +0000 2023-10-21T04:04:30.157 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2023-10-21 03:59:24.534320902 +0000 2023-10-21T04:04:30.157 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2023-10-21 03:59:24.534320902 +0000 2023-10-21T04:04:30.157 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2023-10-21T04:04:30.157 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-21T04:04:30.211 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2023-10-21T04:04:30.212 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2023-10-21T04:04:30.212 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000389733 s, 1.3 MB/s 2023-10-21T04:04:30.213 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-21T04:04:30.263 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2023-10-21T04:04:30.312 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-21T04:04:30.312 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T04:04:30.312 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 933 Links: 1 2023-10-21T04:04:30.313 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T04:04:30.313 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2023-10-21 03:59:25.506312220 +0000 2023-10-21T04:04:30.313 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2023-10-21 03:59:25.134315542 +0000 2023-10-21T04:04:30.313 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2023-10-21 03:59:25.134315542 +0000 2023-10-21T04:04:30.313 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2023-10-21T04:04:30.314 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-21T04:04:30.366 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2023-10-21T04:04:30.367 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2023-10-21T04:04:30.367 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000400456 s, 1.3 MB/s 2023-10-21T04:04:30.368 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-21T04:04:30.417 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T04:04:30.417 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-21T04:04:30.418 INFO:tasks.ceph:Generating config... 2023-10-21T04:04:30.419 INFO:tasks.ceph:[client] rbd default data pool = datapool 2023-10-21T04:04:30.419 INFO:tasks.ceph:[client] rbd default features = 61 2023-10-21T04:04:30.420 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-21T04:04:30.420 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-21T04:04:30.420 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-21T04:04:30.420 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-21T04:04:30.421 INFO:tasks.ceph:[mgr] debug monc = 20 2023-10-21T04:04:30.421 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-21T04:04:30.421 INFO:tasks.ceph:[mgr] debug objecter = 20 2023-10-21T04:04:30.421 INFO:tasks.ceph:[mgr] debug rados = 20 2023-10-21T04:04:30.422 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-21T04:04:30.422 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-21T04:04:30.422 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-21T04:04:30.422 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-21T04:04:30.423 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-21T04:04:30.423 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-21T04:04:30.423 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-10-21T04:04:30.424 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-21T04:04:30.424 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-21T04:04:30.424 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-21T04:04:30.425 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-21T04:04:30.425 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-21T04:04:30.425 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-21T04:04:30.425 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-21T04:04:30.426 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-21T04:04:30.426 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-21T04:04:30.426 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-21T04:04:30.426 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-21T04:04:30.427 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-21T04:04:30.427 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-21T04:04:30.427 INFO:tasks.ceph:Setting up mon.a... 2023-10-21T04:04:30.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-21T04:04:30.490 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/ceph.keyring 2023-10-21T04:04:30.501 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-21T04:04:30.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-21T04:04:30.635 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.119')] 2023-10-21T04:04:30.635 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.119', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 20, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 61}, 'mon.a': {}} 2023-10-21T04:04:30.636 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:30.636 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-21T04:04:30.683 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.119 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:30.763 INFO:teuthology.orchestra.run.smithi119.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:30.763 INFO:teuthology.orchestra.run.smithi119.stdout:monmaptool: generated fsid 57c43eaf-d2a8-4c31-813f-7882b2f380d6 2023-10-21T04:04:30.764 INFO:teuthology.orchestra.run.smithi119.stdout:setting min_mon_release = pacific 2023-10-21T04:04:30.764 INFO:teuthology.orchestra.run.smithi119.stdout:epoch 0 2023-10-21T04:04:30.764 INFO:teuthology.orchestra.run.smithi119.stdout:fsid 57c43eaf-d2a8-4c31-813f-7882b2f380d6 2023-10-21T04:04:30.765 INFO:teuthology.orchestra.run.smithi119.stdout:last_changed 2023-10-21T04:04:30.761939+0000 2023-10-21T04:04:30.765 INFO:teuthology.orchestra.run.smithi119.stdout:created 2023-10-21T04:04:30.761939+0000 2023-10-21T04:04:30.766 INFO:teuthology.orchestra.run.smithi119.stdout:min_mon_release 16 (pacific) 2023-10-21T04:04:30.766 INFO:teuthology.orchestra.run.smithi119.stdout:election_strategy: 1 2023-10-21T04:04:30.766 INFO:teuthology.orchestra.run.smithi119.stdout:0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2023-10-21T04:04:30.767 INFO:teuthology.orchestra.run.smithi119.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-21T04:04:30.768 DEBUG:teuthology.orchestra.run.smithi119:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-21T04:04:30.816 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 57c43eaf-d2a8-4c31-813f-7882b2f380d6... 2023-10-21T04:04:30.817 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-21T04:04:30.889 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2023-10-21T04:04:30.890 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2023-10-21T04:04:30.890 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-21T04:04:30.890 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2023-10-21T04:04:30.891 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.891 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2023-10-21T04:04:30.891 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.891 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2023-10-21T04:04:30.891 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.892 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2023-10-21T04:04:30.892 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2023-10-21T04:04:30.892 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.892 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2023-10-21T04:04:30.892 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2023-10-21T04:04:30.893 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.893 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-21T04:04:30.893 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2023-10-21T04:04:30.893 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.893 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2023-10-21T04:04:30.894 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-21T04:04:30.894 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.894 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2023-10-21T04:04:30.894 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.894 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2023-10-21T04:04:30.895 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2023-10-21T04:04:30.895 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2023-10-21T04:04:30.895 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2023-10-21T04:04:30.895 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2023-10-21T04:04:30.895 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2023-10-21T04:04:30.896 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2023-10-21T04:04:30.896 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-21T04:04:30.896 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-21T04:04:30.896 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2023-10-21T04:04:30.896 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.897 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-21T04:04:30.897 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.897 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2023-10-21T04:04:30.897 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.897 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2023-10-21T04:04:30.898 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.898 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2023-10-21T04:04:30.898 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2023-10-21T04:04:30.898 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2023-10-21T04:04:30.898 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = 172.21.15.119 2023-10-21T04:04:30.899 INFO:teuthology.orchestra.run.smithi119.stdout: mon client directed command retry = 5 2023-10-21T04:04:30.899 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on pool no app = False 2023-10-21T04:04:30.899 INFO:teuthology.orchestra.run.smithi119.stdout: ms inject socket failures = 5000 2023-10-21T04:04:30.899 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = 57c43eaf-d2a8-4c31-813f-7882b2f380d6 2023-10-21T04:04:30.900 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.900 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2023-10-21T04:04:30.900 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2023-10-21T04:04:30.900 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.901 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2023-10-21T04:04:30.901 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2023-10-21T04:04:30.901 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2023-10-21T04:04:30.901 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.901 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2023-10-21T04:04:30.902 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2023-10-21T04:04:30.902 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.902 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2023-10-21T04:04:30.902 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2023-10-21T04:04:30.902 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2023-10-21T04:04:30.903 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.903 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2023-10-21T04:04:30.903 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2023-10-21T04:04:30.903 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.904 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2023-10-21T04:04:30.904 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.904 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2023-10-21T04:04:30.904 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.904 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2023-10-21T04:04:30.905 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.905 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2023-10-21T04:04:30.905 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2023-10-21T04:04:30.905 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.905 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2023-10-21T04:04:30.906 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2023-10-21T04:04:30.906 INFO:teuthology.orchestra.run.smithi119.stdout: bdev async discard = True 2023-10-21T04:04:30.906 INFO:teuthology.orchestra.run.smithi119.stdout: bdev enable discard = True 2023-10-21T04:04:30.906 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore allocator = stupid 2023-10-21T04:04:30.906 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore block size = 96636764160 2023-10-21T04:04:30.907 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore fsck on mount = True 2023-10-21T04:04:30.907 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluefs = 1/20 2023-10-21T04:04:30.907 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluestore = 1/20 2023-10-21T04:04:30.907 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2023-10-21T04:04:30.907 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2023-10-21T04:04:30.908 INFO:teuthology.orchestra.run.smithi119.stdout: debug rocksdb = 4/10 2023-10-21T04:04:30.908 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd backfillfull_ratio = 0.85 2023-10-21T04:04:30.908 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd full ratio = 0.9 2023-10-21T04:04:30.908 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd nearfull ratio = 0.8 2023-10-21T04:04:30.908 INFO:teuthology.orchestra.run.smithi119.stdout: osd failsafe full ratio = 0.95 2023-10-21T04:04:30.909 INFO:teuthology.orchestra.run.smithi119.stdout: osd objectstore = bluestore 2023-10-21T04:04:30.909 INFO:teuthology.orchestra.run.smithi119.stdout: osd shutdown pgref assert = True 2023-10-21T04:04:30.909 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.909 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2023-10-21T04:04:30.909 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2023-10-21T04:04:30.910 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2023-10-21T04:04:30.910 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2023-10-21T04:04:30.910 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2023-10-21T04:04:30.910 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2023-10-21T04:04:30.910 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2023-10-21T04:04:30.911 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2023-10-21T04:04:30.911 INFO:teuthology.orchestra.run.smithi119.stdout: debug monc = 20 2023-10-21T04:04:30.911 INFO:teuthology.orchestra.run.smithi119.stdout: debug objecter = 20 2023-10-21T04:04:30.911 INFO:teuthology.orchestra.run.smithi119.stdout: debug rados = 20 2023-10-21T04:04:30.912 INFO:teuthology.orchestra.run.smithi119.stdout: debug rbd = 20 2023-10-21T04:04:30.912 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.912 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2023-10-21T04:04:30.912 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2023-10-21T04:04:30.912 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2023-10-21T04:04:30.913 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2023-10-21T04:04:30.913 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2023-10-21T04:04:30.913 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2023-10-21T04:04:30.913 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2023-10-21T04:04:30.913 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2023-10-21T04:04:30.913 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2023-10-21T04:04:30.914 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2023-10-21T04:04:30.914 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2023-10-21T04:04:30.914 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.914 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-21T04:04:30.914 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2023-10-21T04:04:30.915 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2023-10-21T04:04:30.915 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.915 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2023-10-21T04:04:30.915 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-21T04:04:30.915 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-21T04:04:30.916 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.916 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2023-10-21T04:04:30.916 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2023-10-21T04:04:30.916 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.916 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2023-10-21T04:04:30.916 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:30.917 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2023-10-21T04:04:30.917 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2023-10-21T04:04:30.917 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2023-10-21T04:04:30.917 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2023-10-21T04:04:30.917 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-21T04:04:30.918 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-21T04:04:30.918 INFO:teuthology.orchestra.run.smithi119.stdout: rbd default data pool = datapool 2023-10-21T04:04:30.918 INFO:teuthology.orchestra.run.smithi119.stdout: rbd default features = 61 2023-10-21T04:04:30.919 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2023-10-21T04:04:30.920 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-21T04:04:30.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-21T04:04:31.013 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-21T04:04:31.014 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:31.014 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-21T04:04:31.021 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:31.021 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-21T04:04:31.068 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi119.front.sepia.ceph.com 2023-10-21T04:04:31.068 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:31.068 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-21T04:04:31.069 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-21T04:04:31.134 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:31.135 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:31.180 INFO:tasks.ceph:Setting up mon nodes... 2023-10-21T04:04:31.180 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-21T04:04:31.181 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-21T04:04:31.265 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-21T04:04:31.268 INFO:tasks.ceph:Setting up mds nodes... 2023-10-21T04:04:31.268 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-21T04:04:31.269 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-21T04:04:31.344 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-21T04:04:31.354 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-21T04:04:31.354 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-21T04:04:31.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-21T04:04:31.411 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T04:04:31.411 INFO:tasks.ceph:role: osd.0 2023-10-21T04:04:31.412 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com 2023-10-21T04:04:31.412 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-21T04:04:31.476 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-21T04:04:31.477 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-21T04:04:31.477 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-21T04:04:31.478 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-21T04:04:31.478 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-21T04:04:31.478 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2023-10-21T04:04:31.479 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-21T04:04:31.479 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-21T04:04:31.479 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-21T04:04:31.480 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-21T04:04:31.928 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2023-10-21T04:04:31.930 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2023-10-21T04:04:31.930 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-21T04:04:32.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-21T04:04:32.037 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: /sbin/restorecon: command not found 2023-10-21T04:04:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:04:32.039 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-21T04:04:32.094 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T04:04:32.094 INFO:tasks.ceph:role: osd.1 2023-10-21T04:04:32.094 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com 2023-10-21T04:04:32.095 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-21T04:04:32.157 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-21T04:04:32.157 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-21T04:04:32.158 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-21T04:04:32.160 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-21T04:04:32.160 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-21T04:04:32.161 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2023-10-21T04:04:32.161 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-21T04:04:32.161 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-21T04:04:32.162 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-21T04:04:32.162 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-21T04:04:32.623 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2023-10-21T04:04:32.624 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2023-10-21T04:04:32.625 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-21T04:04:32.744 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-21T04:04:32.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:04:32.756 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: /sbin/restorecon: command not found 2023-10-21T04:04:32.757 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-21T04:04:32.811 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T04:04:32.811 INFO:tasks.ceph:role: osd.2 2023-10-21T04:04:32.812 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com 2023-10-21T04:04:32.812 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-21T04:04:32.875 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-21T04:04:32.875 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-21T04:04:32.875 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-21T04:04:32.875 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-21T04:04:32.876 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-21T04:04:32.876 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2023-10-21T04:04:32.876 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-21T04:04:32.876 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-21T04:04:32.876 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-21T04:04:32.877 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-21T04:04:33.369 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2023-10-21T04:04:33.371 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2023-10-21T04:04:33.372 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-21T04:04:33.523 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-21T04:04:33.535 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: /sbin/restorecon: command not found 2023-10-21T04:04:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:04:33.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:33.648 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:33.645+0000 7f3a6a24c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-21T04:04:33.649 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:33.645+0000 7f3a6a24c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-21T04:04:33.649 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:33.645+0000 7f3a6a24c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-21T04:04:33.649 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:33.645+0000 7f3a6a24c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-21T04:04:33.649 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:33.645+0000 7f3a6a24c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-21T04:04:33.650 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:33.645+0000 7f3a6a24c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-21T04:04:36.652 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-21T04:04:36.665 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:36.749 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:36.745+0000 7fa22833f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-21T04:04:36.750 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:36.745+0000 7fa22833f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-21T04:04:36.750 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:36.745+0000 7fa22833f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-21T04:04:36.750 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:36.745+0000 7fa22833f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-21T04:04:36.751 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:36.745+0000 7fa22833f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-21T04:04:36.751 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:36.745+0000 7fa22833f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-21T04:04:39.759 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-21T04:04:39.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:39.907 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:39.901+0000 7fecc1f1c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-21T04:04:39.908 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:39.905+0000 7fecc1f1c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-21T04:04:39.908 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:39.905+0000 7fecc1f1c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-21T04:04:39.909 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:39.905+0000 7fecc1f1c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-21T04:04:39.909 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:39.905+0000 7fecc1f1c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-21T04:04:39.909 INFO:teuthology.orchestra.run.smithi119.stderr:2023-10-21T04:04:39.905+0000 7fecc1f1c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-21T04:04:42.911 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-21T04:04:42.968 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-21T04:04:42.968 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:42.969 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-21T04:04:43.022 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:43.023 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-21T04:04:43.075 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:43.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-21T04:04:43.131 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:43.131 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-21T04:04:43.187 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:43.187 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-21T04:04:43.235 INFO:tasks.ceph:Adding keys to all mons... 2023-10-21T04:04:43.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-21T04:04:43.289 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.x] 2023-10-21T04:04:43.290 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDPTTNlxg/ODxAAgKE3BbVEhBDyPglP8as+Fg== 2023-10-21T04:04:43.290 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.0] 2023-10-21T04:04:43.290 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDRTTNlx9WaJhAAtGtb9kDc1usUDyfvnEBEzQ== 2023-10-21T04:04:43.290 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.1] 2023-10-21T04:04:43.291 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDUTTNlC7ijLBAAN4iwbAZqJUUlHEnTlWKiVA== 2023-10-21T04:04:43.291 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.2] 2023-10-21T04:04:43.291 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDXTTNlYmkHNhAAeqOL9Q8hyXmqoS6v+UIYcA== 2023-10-21T04:04:43.291 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2023-10-21T04:04:43.291 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDPTTNl3sh/FBAA0zugArKYaj0lLKqw8le8sQ== 2023-10-21T04:04:43.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-21T04:04:43.379 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-21T04:04:43.468 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-21T04:04:43.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-21T04:04:43.589 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-21T04:04:43.672 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-21T04:04:43.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-21T04:04:43.726 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-21T04:04:43.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-21T04:04:43.833 DEBUG:teuthology.orchestra.run.smithi119:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:04:43.879 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-21T04:04:43.880 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-21T04:04:43.881 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-21T04:04:43.925 INFO:tasks.ceph.mon.a:Started 2023-10-21T04:04:43.925 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-21T04:04:43.925 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-21T04:04:43.926 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-21T04:04:43.928 INFO:tasks.ceph.mgr.x:Started 2023-10-21T04:04:43.928 INFO:tasks.ceph:Setting crush tunables to default 2023-10-21T04:04:43.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd crush tunables default 2023-10-21T04:04:44.171 INFO:teuthology.orchestra.run.smithi119.stderr:adjusted tunables profile to default 2023-10-21T04:04:44.189 INFO:tasks.ceph:check_enable_crimson: False 2023-10-21T04:04:44.189 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-21T04:04:44.190 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:44.190 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-21T04:04:44.200 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:44.201 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-21T04:04:44.252 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:04:44.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-21T04:04:44.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 7bb86598-0c9a-42d0-89ca-b2506175e59d 0 2023-10-21T04:04:44.321 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:44.317+0000 7ffb3227e180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-21T04:04:44.519 INFO:teuthology.orchestra.run.smithi119.stdout:0 2023-10-21T04:04:44.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new c8b63612-e8a0-4b1b-bf22-e4befb5787b2 1 2023-10-21T04:04:44.726 INFO:teuthology.orchestra.run.smithi119.stdout:1 2023-10-21T04:04:44.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new bdf66e75-d9b7-40fe-bf4d-67a17adc0a12 2 2023-10-21T04:04:44.867 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:44.865+0000 7ffb3227e180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-21T04:04:44.930 INFO:teuthology.orchestra.run.smithi119.stdout:2 2023-10-21T04:04:44.946 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-21T04:04:44.946 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-21T04:04:44.948 INFO:tasks.ceph.osd.0:Started 2023-10-21T04:04:44.949 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-21T04:04:44.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-21T04:04:44.951 INFO:tasks.ceph.osd.1:Started 2023-10-21T04:04:44.951 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-21T04:04:44.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-21T04:04:44.954 INFO:tasks.ceph.osd.2:Started 2023-10-21T04:04:44.954 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T04:04:45.067 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:45.065+0000 7ffb3227e180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-21T04:04:45.137 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:04:45.137 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":5,"fsid":"57c43eaf-d2a8-4c31-813f-7882b2f380d6","created":"2023-10-21T04:04:44.062479+0000","modified":"2023-10-21T04:04:44.928236+0000","last_up_change":"0.000000","last_in_change":"2023-10-21T04:04:44.928236+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bb86598-0c9a-42d0-89ca-b2506175e59d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c8b63612-e8a0-4b1b-bf22-e4befb5787b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bdf66e75-d9b7-40fe-bf4d-67a17adc0a12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:04:45.153 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-21T04:04:45.153 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-21T04:04:45.455 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-21T04:04:45.568 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:04:45.565+0000 7fca05209640 -1 Falling back to public interface 2023-10-21T04:04:45.588 INFO:tasks.ceph.osd.2.smithi119.stderr:2023-10-21T04:04:45.585+0000 7f41db3a3640 -1 Falling back to public interface 2023-10-21T04:04:45.596 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:04:45.593+0000 7f875bdc7640 -1 Falling back to public interface 2023-10-21T04:04:45.638 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:45.633+0000 7ffb3227e180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-21T04:04:45.640 INFO:teuthology.misc.health.smithi119.stdout: 2023-10-21T04:04:45.641 INFO:teuthology.misc.health.smithi119.stdout:{"epoch":5,"fsid":"57c43eaf-d2a8-4c31-813f-7882b2f380d6","created":"2023-10-21T04:04:44.062479+0000","modified":"2023-10-21T04:04:44.928236+0000","last_up_change":"0.000000","last_in_change":"2023-10-21T04:04:44.928236+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bb86598-0c9a-42d0-89ca-b2506175e59d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c8b63612-e8a0-4b1b-bf22-e4befb5787b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bdf66e75-d9b7-40fe-bf4d-67a17adc0a12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:04:45.655 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-21T04:04:45.693 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:45.689+0000 7ffb3227e180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-21T04:04:45.746 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:45.741+0000 7ffb3227e180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-21T04:04:45.822 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:45.817+0000 7ffb3227e180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-21T04:04:45.874 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:45.869+0000 7ffb3227e180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-21T04:04:45.951 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:45.949+0000 7ffb3227e180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-21T04:04:46.002 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:45.997+0000 7ffb3227e180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-21T04:04:46.141 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:46.137+0000 7ffb3227e180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-21T04:04:46.202 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:46.197+0000 7ffb3227e180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-21T04:04:46.852 INFO:tasks.ceph.mgr.x.smithi119.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-21T04:04:46.853 INFO:tasks.ceph.mgr.x.smithi119.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-21T04:04:46.853 INFO:tasks.ceph.mgr.x.smithi119.stderr: from numpy import show_config as show_numpy_config 2023-10-21T04:04:46.912 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:46.909+0000 7ffb3227e180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-21T04:04:47.058 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:47.053+0000 7ffb3227e180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-21T04:04:47.230 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:47.225+0000 7ffb3227e180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-21T04:04:47.300 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:47.297+0000 7ffb3227e180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-21T04:04:47.436 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:47.433+0000 7ffb3227e180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-21T04:04:48.048 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:48.045+0000 7ffb3227e180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-21T04:04:48.126 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:48.121+0000 7ffb3227e180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-21T04:04:48.312 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:48.309+0000 7ffb3227e180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-21T04:04:48.388 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:48.385+0000 7ffb3227e180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-21T04:04:48.701 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:48.697+0000 7ffb3227e180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-21T04:04:48.761 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:48.757+0000 7ffb3227e180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-21T04:04:49.223 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:04:49.221+0000 7f875bdc7640 -1 osd.1 0 log_to_monitors true 2023-10-21T04:04:49.429 INFO:tasks.ceph.osd.2.smithi119.stderr:2023-10-21T04:04:49.425+0000 7f41db3a3640 -1 osd.2 0 log_to_monitors true 2023-10-21T04:04:49.429 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:04:49.425+0000 7fca05209640 -1 osd.0 0 log_to_monitors true 2023-10-21T04:04:51.865 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:04:51.861+0000 7ffad9af2640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-21T04:04:51.958 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-21T04:04:52.374 INFO:teuthology.misc.health.smithi119.stdout: 2023-10-21T04:04:52.374 INFO:teuthology.misc.health.smithi119.stdout:{"epoch":7,"fsid":"57c43eaf-d2a8-4c31-813f-7882b2f380d6","created":"2023-10-21T04:04:44.062479+0000","modified":"2023-10-21T04:04:51.015244+0000","last_up_change":"0.000000","last_in_change":"2023-10-21T04:04:44.928236+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bb86598-0c9a-42d0-89ca-b2506175e59d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c8b63612-e8a0-4b1b-bf22-e4befb5787b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bdf66e75-d9b7-40fe-bf4d-67a17adc0a12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:04:52.399 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-21T04:04:53.018 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:04:53.013+0000 7f8758d84640 -1 osd.1 0 waiting for initial osdmap 2023-10-21T04:04:53.036 INFO:tasks.ceph.osd.2.smithi119.stderr:2023-10-21T04:04:53.033+0000 7f41d6b28640 -1 osd.2 0 waiting for initial osdmap 2023-10-21T04:04:53.043 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:04:53.041+0000 7f874fb93640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-21T04:04:53.046 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:04:53.041+0000 7fca0098e640 -1 osd.0 0 waiting for initial osdmap 2023-10-21T04:04:53.063 INFO:tasks.ceph.osd.2.smithi119.stderr:2023-10-21T04:04:53.061+0000 7f41cf16f640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-21T04:04:53.079 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:04:53.077+0000 7fc9f8fd5640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-21T04:04:58.701 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-21T04:04:58.990 INFO:teuthology.misc.health.smithi119.stdout: 2023-10-21T04:04:58.991 INFO:teuthology.misc.health.smithi119.stdout:{"epoch":11,"fsid":"57c43eaf-d2a8-4c31-813f-7882b2f380d6","created":"2023-10-21T04:04:44.062479+0000","modified":"2023-10-21T04:04:57.143806+0000","last_up_change":"2023-10-21T04:04:54.047207+0000","last_in_change":"2023-10-21T04:04:44.928236+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T04:04:54.882624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7bb86598-0c9a-42d0-89ca-b2506175e59d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6801","nonce":70176460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6803","nonce":70176460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6807","nonce":70176460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6805","nonce":70176460}]},"public_addr":"172.21.15.119:6801/70176460","cluster_addr":"172.21.15.119:6803/70176460","heartbeat_back_addr":"172.21.15.119:6807/70176460","heartbeat_front_addr":"172.21.15.119:6805/70176460","state":["exists","up"]},{"osd":1,"uuid":"c8b63612-e8a0-4b1b-bf22-e4befb5787b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6817","nonce":2575111078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6819","nonce":2575111078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6823","nonce":2575111078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6821","nonce":2575111078}]},"public_addr":"172.21.15.119:6817/2575111078","cluster_addr":"172.21.15.119:6819/2575111078","heartbeat_back_addr":"172.21.15.119:6823/2575111078","heartbeat_front_addr":"172.21.15.119:6821/2575111078","state":["exists","up"]},{"osd":2,"uuid":"bdf66e75-d9b7-40fe-bf4d-67a17adc0a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6809","nonce":1848266345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6811","nonce":1848266345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6815","nonce":1848266345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6813","nonce":1848266345}]},"public_addr":"172.21.15.119:6809/1848266345","cluster_addr":"172.21.15.119:6811/1848266345","heartbeat_back_addr":"172.21.15.119:6815/1848266345","heartbeat_front_addr":"172.21.15.119:6813/1848266345","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:04:50.447383+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:04:50.187584+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:04:50.446482+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:04:59.007 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-21T04:04:59.008 INFO:tasks.ceph:Creating RBD pool 2023-10-21T04:04:59.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-21T04:05:00.168 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'rbd' created 2023-10-21T04:05:00.195 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-21T04:05:01.174 INFO:teuthology.orchestra.run.smithi119.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-21T04:05:01.205 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-21T04:05:01.206 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-21T04:05:01.206 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-21T04:05:01.206 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-21T04:05:01.207 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-21T04:05:01.529 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:05:01.545 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1705553833},{"type":"v1","addr":"172.21.15.119:6825","nonce":1705553833}]},"active_addr":"172.21.15.119:6825/1705553833","active_change":"2023-10-21T04:04:50.801401+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":4268465796}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":1632625295}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":2444104118}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":3574829714}]}]} 2023-10-21T04:05:01.548 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-21T04:05:01.548 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-21T04:05:01.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T04:05:01.812 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:05:01.812 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":13,"fsid":"57c43eaf-d2a8-4c31-813f-7882b2f380d6","created":"2023-10-21T04:04:44.062479+0000","modified":"2023-10-21T04:05:01.168096+0000","last_up_change":"2023-10-21T04:04:54.047207+0000","last_in_change":"2023-10-21T04:04:44.928236+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T04:04:54.882624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-21T04:04:59.332315+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7bb86598-0c9a-42d0-89ca-b2506175e59d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6801","nonce":70176460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6803","nonce":70176460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6807","nonce":70176460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6805","nonce":70176460}]},"public_addr":"172.21.15.119:6801/70176460","cluster_addr":"172.21.15.119:6803/70176460","heartbeat_back_addr":"172.21.15.119:6807/70176460","heartbeat_front_addr":"172.21.15.119:6805/70176460","state":["exists","up"]},{"osd":1,"uuid":"c8b63612-e8a0-4b1b-bf22-e4befb5787b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6817","nonce":2575111078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6819","nonce":2575111078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6823","nonce":2575111078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6821","nonce":2575111078}]},"public_addr":"172.21.15.119:6817/2575111078","cluster_addr":"172.21.15.119:6819/2575111078","heartbeat_back_addr":"172.21.15.119:6823/2575111078","heartbeat_front_addr":"172.21.15.119:6821/2575111078","state":["exists","up"]},{"osd":2,"uuid":"bdf66e75-d9b7-40fe-bf4d-67a17adc0a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6809","nonce":1848266345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6811","nonce":1848266345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6815","nonce":1848266345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6813","nonce":1848266345}]},"public_addr":"172.21.15.119:6809/1848266345","cluster_addr":"172.21.15.119:6811/1848266345","heartbeat_back_addr":"172.21.15.119:6815/1848266345","heartbeat_front_addr":"172.21.15.119:6813/1848266345","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:04:50.447383+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:04:50.187584+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:04:50.446482+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:05:01.828 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-21T04:05:01.829 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T04:05:02.102 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:05:02.103 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":13,"fsid":"57c43eaf-d2a8-4c31-813f-7882b2f380d6","created":"2023-10-21T04:04:44.062479+0000","modified":"2023-10-21T04:05:01.168096+0000","last_up_change":"2023-10-21T04:04:54.047207+0000","last_in_change":"2023-10-21T04:04:44.928236+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T04:04:54.882624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-21T04:04:59.332315+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7bb86598-0c9a-42d0-89ca-b2506175e59d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6801","nonce":70176460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6803","nonce":70176460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6807","nonce":70176460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6805","nonce":70176460}]},"public_addr":"172.21.15.119:6801/70176460","cluster_addr":"172.21.15.119:6803/70176460","heartbeat_back_addr":"172.21.15.119:6807/70176460","heartbeat_front_addr":"172.21.15.119:6805/70176460","state":["exists","up"]},{"osd":1,"uuid":"c8b63612-e8a0-4b1b-bf22-e4befb5787b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6817","nonce":2575111078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6819","nonce":2575111078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6823","nonce":2575111078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6821","nonce":2575111078}]},"public_addr":"172.21.15.119:6817/2575111078","cluster_addr":"172.21.15.119:6819/2575111078","heartbeat_back_addr":"172.21.15.119:6823/2575111078","heartbeat_front_addr":"172.21.15.119:6821/2575111078","state":["exists","up"]},{"osd":2,"uuid":"bdf66e75-d9b7-40fe-bf4d-67a17adc0a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6809","nonce":1848266345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6811","nonce":1848266345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6815","nonce":1848266345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6813","nonce":1848266345}]},"public_addr":"172.21.15.119:6809/1848266345","cluster_addr":"172.21.15.119:6811/1848266345","heartbeat_back_addr":"172.21.15.119:6815/1848266345","heartbeat_front_addr":"172.21.15.119:6813/1848266345","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:04:50.447383+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:04:50.187584+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:04:50.446482+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:05:02.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-21T04:05:02.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-21T04:05:02.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-21T04:05:02.267 INFO:teuthology.orchestra.run.smithi119.stdout:34359738374 2023-10-21T04:05:02.268 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-21T04:05:02.283 INFO:teuthology.orchestra.run.smithi119.stdout:34359738373 2023-10-21T04:05:02.283 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-21T04:05:02.286 INFO:teuthology.orchestra.run.smithi119.stdout:34359738373 2023-10-21T04:05:02.286 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-21T04:05:02.545 INFO:teuthology.orchestra.run.smithi119.stdout:34359738372 2023-10-21T04:05:02.562 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738372 for osd.0 2023-10-21T04:05:02.600 INFO:teuthology.orchestra.run.smithi119.stdout:34359738372 2023-10-21T04:05:02.608 INFO:teuthology.orchestra.run.smithi119.stdout:34359738372 2023-10-21T04:05:02.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-10-21T04:05:02.626 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-10-21T04:05:03.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-21T04:05:03.619 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-21T04:05:03.626 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-21T04:05:03.864 INFO:teuthology.orchestra.run.smithi119.stdout:34359738374 2023-10-21T04:05:03.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2023-10-21T04:05:03.882 DEBUG:teuthology.parallel:result is None 2023-10-21T04:05:03.904 INFO:teuthology.orchestra.run.smithi119.stdout:34359738373 2023-10-21T04:05:03.925 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-10-21T04:05:03.926 DEBUG:teuthology.parallel:result is None 2023-10-21T04:05:03.954 INFO:teuthology.orchestra.run.smithi119.stdout:34359738373 2023-10-21T04:05:03.971 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-10-21T04:05:03.971 DEBUG:teuthology.parallel:result is None 2023-10-21T04:05:03.971 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-21T04:05:03.972 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T04:05:04.285 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:05:04.286 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T04:05:04.302 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-10-21T04:05:02.812465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82556,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032964,"statfs":{"total":289910292480,"available":289825755136,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.643814"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:01.191492+0000","last_change":"2023-10-21T04:05:01.187428+0000","last_active":"2023-10-21T04:05:01.191492+0000","last_peered":"2023-10-21T04:05:01.191492+0000","last_clean":"2023-10-21T04:05:01.191492+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T04:05:01.191492+0000","last_undegraded":"2023-10-21T04:05:01.191492+0000","last_fullsized":"2023-10-21T04:05:01.191492+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:31.855193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:01.191517+0000","last_change":"2023-10-21T04:05:01.187482+0000","last_active":"2023-10-21T04:05:01.191517+0000","last_peered":"2023-10-21T04:05:01.191517+0000","last_clean":"2023-10-21T04:05:01.191517+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T04:05:01.191517+0000","last_undegraded":"2023-10-21T04:05:01.191517+0000","last_fullsized":"2023-10-21T04:05:01.191517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:10:35.750383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:01.191539+0000","last_change":"2023-10-21T04:05:01.187628+0000","last_active":"2023-10-21T04:05:01.191539+0000","last_peered":"2023-10-21T04:05:01.191539+0000","last_clean":"2023-10-21T04:05:01.191539+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T04:05:01.191539+0000","last_undegraded":"2023-10-21T04:05:01.191539+0000","last_fullsized":"2023-10-21T04:05:01.191539+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:06:27.276306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:01.191595+0000","last_change":"2023-10-21T04:05:01.188633+0000","last_active":"2023-10-21T04:05:01.191595+0000","last_peered":"2023-10-21T04:05:01.191595+0000","last_clean":"2023-10-21T04:05:01.191595+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T04:05:01.191595+0000","last_undegraded":"2023-10-21T04:05:01.191595+0000","last_fullsized":"2023-10-21T04:05:01.191595+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:54:35.925036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:01.187428+0000","last_change":"2023-10-21T04:05:01.187428+0000","last_active":"2023-10-21T04:05:01.187428+0000","last_peered":"2023-10-21T04:05:01.187428+0000","last_clean":"2023-10-21T04:05:01.187428+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T04:05:01.187428+0000","last_undegraded":"2023-10-21T04:05:01.187428+0000","last_fullsized":"2023-10-21T04:05:01.187428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:02:39.078325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:05:02.181180+0000","last_change":"2023-10-21T04:05:01.189879+0000","last_active":"2023-10-21T04:05:02.181180+0000","last_peered":"2023-10-21T04:05:02.181180+0000","last_clean":"2023-10-21T04:05:02.181180+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T04:05:02.181180+0000","last_undegraded":"2023-10-21T04:05:02.181180+0000","last_fullsized":"2023-10-21T04:05:02.181180+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:41:43.561612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:05:02.181165+0000","last_change":"2023-10-21T04:05:01.190036+0000","last_active":"2023-10-21T04:05:02.181165+0000","last_peered":"2023-10-21T04:05:02.181165+0000","last_clean":"2023-10-21T04:05:02.181165+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T04:05:02.181165+0000","last_undegraded":"2023-10-21T04:05:02.181165+0000","last_fullsized":"2023-10-21T04:05:02.181165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:52:35.513076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:01.191435+0000","last_change":"2023-10-21T04:05:01.186422+0000","last_active":"2023-10-21T04:05:01.191435+0000","last_peered":"2023-10-21T04:05:01.191435+0000","last_clean":"2023-10-21T04:05:01.191435+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T04:05:01.191435+0000","last_undegraded":"2023-10-21T04:05:01.191435+0000","last_fullsized":"2023-10-21T04:05:01.191435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:04:16.637120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-21T04:05:00.177988+0000","last_change":"2023-10-21T04:04:56.457261+0000","last_active":"2023-10-21T04:05:00.177988+0000","last_peered":"2023-10-21T04:05:00.177988+0000","last_clean":"2023-10-21T04:05:00.177988+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T04:05:00.177988+0000","last_undegraded":"2023-10-21T04:05:00.177988+0000","last_fullsized":"2023-10-21T04:05:00.177988+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:04:55.099596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:04:55.099596+0000","last_clean_scrub_stamp":"2023-10-21T04:04:55.099596+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:20:18.949598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T04:05:04.303 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T04:05:04.566 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:05:04.567 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T04:05:04.582 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-10-21T04:05:02.812465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82556,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032964,"statfs":{"total":289910292480,"available":289825755136,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.643814"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:01.191492+0000","last_change":"2023-10-21T04:05:01.187428+0000","last_active":"2023-10-21T04:05:01.191492+0000","last_peered":"2023-10-21T04:05:01.191492+0000","last_clean":"2023-10-21T04:05:01.191492+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T04:05:01.191492+0000","last_undegraded":"2023-10-21T04:05:01.191492+0000","last_fullsized":"2023-10-21T04:05:01.191492+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:31.855193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:01.191517+0000","last_change":"2023-10-21T04:05:01.187482+0000","last_active":"2023-10-21T04:05:01.191517+0000","last_peered":"2023-10-21T04:05:01.191517+0000","last_clean":"2023-10-21T04:05:01.191517+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T04:05:01.191517+0000","last_undegraded":"2023-10-21T04:05:01.191517+0000","last_fullsized":"2023-10-21T04:05:01.191517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:10:35.750383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:01.191539+0000","last_change":"2023-10-21T04:05:01.187628+0000","last_active":"2023-10-21T04:05:01.191539+0000","last_peered":"2023-10-21T04:05:01.191539+0000","last_clean":"2023-10-21T04:05:01.191539+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T04:05:01.191539+0000","last_undegraded":"2023-10-21T04:05:01.191539+0000","last_fullsized":"2023-10-21T04:05:01.191539+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:06:27.276306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:01.191595+0000","last_change":"2023-10-21T04:05:01.188633+0000","last_active":"2023-10-21T04:05:01.191595+0000","last_peered":"2023-10-21T04:05:01.191595+0000","last_clean":"2023-10-21T04:05:01.191595+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T04:05:01.191595+0000","last_undegraded":"2023-10-21T04:05:01.191595+0000","last_fullsized":"2023-10-21T04:05:01.191595+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:54:35.925036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:01.187428+0000","last_change":"2023-10-21T04:05:01.187428+0000","last_active":"2023-10-21T04:05:01.187428+0000","last_peered":"2023-10-21T04:05:01.187428+0000","last_clean":"2023-10-21T04:05:01.187428+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T04:05:01.187428+0000","last_undegraded":"2023-10-21T04:05:01.187428+0000","last_fullsized":"2023-10-21T04:05:01.187428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:02:39.078325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:05:02.181180+0000","last_change":"2023-10-21T04:05:01.189879+0000","last_active":"2023-10-21T04:05:02.181180+0000","last_peered":"2023-10-21T04:05:02.181180+0000","last_clean":"2023-10-21T04:05:02.181180+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T04:05:02.181180+0000","last_undegraded":"2023-10-21T04:05:02.181180+0000","last_fullsized":"2023-10-21T04:05:02.181180+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:41:43.561612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:05:02.181165+0000","last_change":"2023-10-21T04:05:01.190036+0000","last_active":"2023-10-21T04:05:02.181165+0000","last_peered":"2023-10-21T04:05:02.181165+0000","last_clean":"2023-10-21T04:05:02.181165+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T04:05:02.181165+0000","last_undegraded":"2023-10-21T04:05:02.181165+0000","last_fullsized":"2023-10-21T04:05:02.181165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:52:35.513076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:05:01.191435+0000","last_change":"2023-10-21T04:05:01.186422+0000","last_active":"2023-10-21T04:05:01.191435+0000","last_peered":"2023-10-21T04:05:01.191435+0000","last_clean":"2023-10-21T04:05:01.191435+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T04:05:01.191435+0000","last_undegraded":"2023-10-21T04:05:01.191435+0000","last_fullsized":"2023-10-21T04:05:01.191435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:04:16.637120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-21T04:05:00.177988+0000","last_change":"2023-10-21T04:04:56.457261+0000","last_active":"2023-10-21T04:05:00.177988+0000","last_peered":"2023-10-21T04:05:00.177988+0000","last_clean":"2023-10-21T04:05:00.177988+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T04:05:00.177988+0000","last_undegraded":"2023-10-21T04:05:00.177988+0000","last_fullsized":"2023-10-21T04:05:00.177988+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:04:55.099596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:04:55.099596+0000","last_clean_scrub_stamp":"2023-10-21T04:04:55.099596+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:20:18.949598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T04:05:04.583 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-21T04:05:04.584 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-21T04:05:04.584 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-21T04:05:04.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-21T04:05:04.873 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T04:05:04.873 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-21T04:05:04.890 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-21T04:05:04.890 INFO:teuthology.run_tasks:Running task exec... 2023-10-21T04:05:04.900 INFO:teuthology.task.exec:Executing custom commands... 2023-10-21T04:05:04.900 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi119.front.sepia.ceph.com 2023-10-21T04:05:04.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4' 2023-10-21T04:05:05.277 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'datapool' created 2023-10-21T04:05:05.303 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2023-10-21T04:05:08.329 INFO:teuthology.run_tasks:Running task install... 2023-10-21T04:05:08.338 DEBUG:teuthology.task.install:project ceph 2023-10-21T04:05:08.338 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}} 2023-10-21T04:05:08.339 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'} 2023-10-21T04:05:08.339 INFO:teuthology.task.install:Using flavor: default 2023-10-21T04:05:08.344 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-21T04:05:08.345 INFO:teuthology.task.install:extra packages: [] 2023-10-21T04:05:08.345 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-key list | grep Ceph 2023-10-21T04:05:08.409 INFO:teuthology.orchestra.run.smithi119.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-21T04:05:08.438 INFO:teuthology.orchestra.run.smithi119.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-21T04:05:08.438 INFO:teuthology.orchestra.run.smithi119.stdout:uid [ unknown] Ceph.com (release key) 2023-10-21T04:05:08.439 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-21T04:05:08.439 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2023-10-21T04:05:08.439 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:05:08.669 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default/ 2023-10-21T04:05:08.669 INFO:teuthology.task.install.deb:Package version is 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:05:08.841 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:05:08.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-21T04:05:08.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get update 2023-10-21T04:05:09.080 INFO:teuthology.orchestra.run.smithi119.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy InRelease 2023-10-21T04:05:09.101 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:2 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release 2023-10-21T04:05:09.124 INFO:teuthology.orchestra.run.smithi119.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-21T04:05:09.178 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-21T04:05:09.178 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-21T04:05:09.248 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-21T04:05:09.330 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-21T04:05:10.433 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2023-10-21T04:05:10.447 DEBUG:teuthology.orchestra.run.smithi119:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6744-g7f103c7c-1jammy cephadm=18.0.0-6744-g7f103c7c-1jammy ceph-mds=18.0.0-6744-g7f103c7c-1jammy ceph-mgr=18.0.0-6744-g7f103c7c-1jammy ceph-common=18.0.0-6744-g7f103c7c-1jammy ceph-fuse=18.0.0-6744-g7f103c7c-1jammy ceph-test=18.0.0-6744-g7f103c7c-1jammy ceph-volume=18.0.0-6744-g7f103c7c-1jammy radosgw=18.0.0-6744-g7f103c7c-1jammy python3-rados=18.0.0-6744-g7f103c7c-1jammy python3-rgw=18.0.0-6744-g7f103c7c-1jammy python3-cephfs=18.0.0-6744-g7f103c7c-1jammy python3-rbd=18.0.0-6744-g7f103c7c-1jammy libcephfs2=18.0.0-6744-g7f103c7c-1jammy libcephfs-dev=18.0.0-6744-g7f103c7c-1jammy librados2=18.0.0-6744-g7f103c7c-1jammy librbd1=18.0.0-6744-g7f103c7c-1jammy rbd-fuse=18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:05:10.509 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2023-10-21T04:05:10.692 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2023-10-21T04:05:10.693 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2023-10-21T04:05:10.856 INFO:teuthology.orchestra.run.smithi119.stdout:ceph is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.857 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-common is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.857 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-fuse is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.857 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-mds is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.857 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-mgr is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.858 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-test is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.858 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-volume is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.858 INFO:teuthology.orchestra.run.smithi119.stdout:cephadm is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.858 INFO:teuthology.orchestra.run.smithi119.stdout:libcephfs-dev is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.858 INFO:teuthology.orchestra.run.smithi119.stdout:libcephfs2 is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.859 INFO:teuthology.orchestra.run.smithi119.stdout:librados2 is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.859 INFO:teuthology.orchestra.run.smithi119.stdout:librbd1 is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.859 INFO:teuthology.orchestra.run.smithi119.stdout:python3-cephfs is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.859 INFO:teuthology.orchestra.run.smithi119.stdout:python3-rados is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.859 INFO:teuthology.orchestra.run.smithi119.stdout:python3-rbd is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.860 INFO:teuthology.orchestra.run.smithi119.stdout:python3-rgw is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.860 INFO:teuthology.orchestra.run.smithi119.stdout:radosgw is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.860 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-fuse is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:05:10.860 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T04:05:10.860 INFO:teuthology.orchestra.run.smithi119.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-21T04:05:10.861 INFO:teuthology.orchestra.run.smithi119.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-21T04:05:10.861 INFO:teuthology.orchestra.run.smithi119.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-21T04:05:10.861 INFO:teuthology.orchestra.run.smithi119.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-21T04:05:10.861 INFO:teuthology.orchestra.run.smithi119.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-21T04:05:10.861 INFO:teuthology.orchestra.run.smithi119.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T04:05:10.894 INFO:teuthology.orchestra.run.smithi119.stdout:0 upgraded, 0 newly installed, 0 to remove and 257 not upgraded. 2023-10-21T04:05:10.895 INFO:teuthology.orchestra.run.smithi119.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-21T04:05:10.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2023-10-21T04:05:10.957 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2023-10-21T04:05:11.161 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2023-10-21T04:05:11.162 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2023-10-21T04:05:11.325 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T04:05:11.325 INFO:teuthology.orchestra.run.smithi119.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-10-21T04:05:11.325 INFO:teuthology.orchestra.run.smithi119.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-10-21T04:05:11.325 INFO:teuthology.orchestra.run.smithi119.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-10-21T04:05:11.326 INFO:teuthology.orchestra.run.smithi119.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-10-21T04:05:11.327 INFO:teuthology.orchestra.run.smithi119.stdout: python2.7 python2.7-dev python2.7-minimal 2023-10-21T04:05:11.327 INFO:teuthology.orchestra.run.smithi119.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T04:05:11.327 INFO:teuthology.orchestra.run.smithi119.stdout:The following additional packages will be installed: 2023-10-21T04:05:11.327 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemblk1 2023-10-21T04:05:11.328 INFO:teuthology.orchestra.run.smithi119.stdout:Suggested packages: 2023-10-21T04:05:11.328 INFO:teuthology.orchestra.run.smithi119.stdout: gnuplot gfio python-scipy 2023-10-21T04:05:11.352 INFO:teuthology.orchestra.run.smithi119.stdout:The following NEW packages will be installed: 2023-10-21T04:05:11.353 INFO:teuthology.orchestra.run.smithi119.stdout: fio libpmemblk1 2023-10-21T04:05:11.533 INFO:teuthology.orchestra.run.smithi119.stdout:0 upgraded, 2 newly installed, 0 to remove and 257 not upgraded. 2023-10-21T04:05:11.533 INFO:teuthology.orchestra.run.smithi119.stdout:Need to get 4,112 kB of archives. 2023-10-21T04:05:11.533 INFO:teuthology.orchestra.run.smithi119.stdout:After this operation, 7,813 kB of additional disk space will be used. 2023-10-21T04:05:11.534 INFO:teuthology.orchestra.run.smithi119.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-10-21T04:05:11.818 INFO:teuthology.orchestra.run.smithi119.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-10-21T04:05:12.681 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 4,112 kB in 1s (4,270 kB/s) 2023-10-21T04:05:12.876 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-10-21T04:05:12.980 INFO:teuthology.orchestra.run.smithi119.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172810 files and directories currently installed.) 2023-10-21T04:05:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-10-21T04:05:13.066 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-21T04:05:13.336 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package fio. 2023-10-21T04:05:13.364 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2023-10-21T04:05:13.403 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking fio (3.28-1) ... 2023-10-21T04:05:13.904 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-21T04:05:14.033 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up fio (3.28-1) ... 2023-10-21T04:05:14.562 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-21T04:05:15.595 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-21T04:05:16.037 INFO:teuthology.orchestra.run.smithi119.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-21T04:05:16.039 DEBUG:teuthology.parallel:result is None 2023-10-21T04:05:16.040 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:05:16.262 DEBUG:teuthology.orchestra.run.smithi119:> dpkg-query -W -f '${Version}' ceph 2023-10-21T04:05:16.282 INFO:teuthology.orchestra.run.smithi119.stdout:18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:05:16.282 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:05:16.282 INFO:teuthology.task.install:The correct ceph version 18.0.0-6744-g7f103c7c-1jammy is installed. 2023-10-21T04:05:16.283 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-21T04:05:16.283 INFO:teuthology.run_tasks:Running task workunit... 2023-10-21T04:05:16.296 INFO:tasks.workunit:Pulling workunits from ref 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:05:16.296 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-21T04:05:16.296 DEBUG:teuthology.orchestra.run.smithi119:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-21T04:05:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:05:16.328 INFO:teuthology.orchestra.run.smithi119.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-21T04:05:16.329 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-21T04:05:16.375 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-21T04:05:16.376 DEBUG:teuthology.orchestra.run.smithi119:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-21T04:05:16.423 INFO:tasks.workunit:timeout=3h 2023-10-21T04:05:16.424 INFO:tasks.workunit:cleanup=True 2023-10-21T04:05:16.424 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:05:16.473 INFO:tasks.workunit.client.0.smithi119.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-21T04:06:18.590 INFO:tasks.workunit.client.0.smithi119.stderr:Note: switching to '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'. 2023-10-21T04:06:18.590 INFO:tasks.workunit.client.0.smithi119.stderr: 2023-10-21T04:06:18.590 INFO:tasks.workunit.client.0.smithi119.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-21T04:06:18.591 INFO:tasks.workunit.client.0.smithi119.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-21T04:06:18.591 INFO:tasks.workunit.client.0.smithi119.stderr:state without impacting any branches by switching back to a branch. 2023-10-21T04:06:18.591 INFO:tasks.workunit.client.0.smithi119.stderr: 2023-10-21T04:06:18.591 INFO:tasks.workunit.client.0.smithi119.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-21T04:06:18.591 INFO:tasks.workunit.client.0.smithi119.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-21T04:06:18.592 INFO:tasks.workunit.client.0.smithi119.stderr: 2023-10-21T04:06:18.592 INFO:tasks.workunit.client.0.smithi119.stderr: git switch -c 2023-10-21T04:06:18.592 INFO:tasks.workunit.client.0.smithi119.stderr: 2023-10-21T04:06:18.592 INFO:tasks.workunit.client.0.smithi119.stderr:Or undo this operation with: 2023-10-21T04:06:18.592 INFO:tasks.workunit.client.0.smithi119.stderr: 2023-10-21T04:06:18.593 INFO:tasks.workunit.client.0.smithi119.stderr: git switch - 2023-10-21T04:06:18.593 INFO:tasks.workunit.client.0.smithi119.stderr: 2023-10-21T04:06:18.593 INFO:tasks.workunit.client.0.smithi119.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-21T04:06:18.593 INFO:tasks.workunit.client.0.smithi119.stderr: 2023-10-21T04:06:18.593 INFO:tasks.workunit.client.0.smithi119.stderr:HEAD is now at 7f103c7cad2 qa/suites/rbd/cli/workloads/rbd_support_module_recovery: more debug 2023-10-21T04:06:18.595 DEBUG:teuthology.orchestra.run.smithi119:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-21T04:06:18.644 INFO:tasks.workunit.client.0.smithi119.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-21T04:06:18.647 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-21T04:06:18.647 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-21T04:06:19.429 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-21T04:06:19.501 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-21T04:06:19.546 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-21T04:06:19.548 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-21T04:06:19.548 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-21T04:06:19.587 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-21T04:06:19.591 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-10-21T04:06:19.591 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-21T04:06:19.638 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-21T04:06:19.639 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-21T04:06:19.640 DEBUG:teuthology.orchestra.run.smithi119:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-21T04:06:19.691 INFO:tasks.workunit.client.0.smithi119.stderr:+ POOL=rbd 2023-10-21T04:06:19.691 INFO:tasks.workunit.client.0.smithi119.stderr:+ IMAGE_PREFIX=image 2023-10-21T04:06:19.691 INFO:tasks.workunit.client.0.smithi119.stderr:+ NUM_IMAGES=20 2023-10-21T04:06:19.691 INFO:tasks.workunit.client.0.smithi119.stderr:+ RUN_TIME=3600 2023-10-21T04:06:19.692 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror pool enable rbd image 2023-10-21T04:06:19.738 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-21T04:06:19.773 INFO:tasks.workunit.client.0.smithi119.stdout:5ac156c2-530f-4574-aad9-b729b627a310 2023-10-21T04:06:19.776 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i = 1 )) 2023-10-21T04:06:19.777 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:19.777 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image1 2023-10-21T04:06:19.827 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-21T04:06:20.145 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:20.158 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-21T04:06:20.219 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:20.219 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:20.219 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image2 2023-10-21T04:06:20.268 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-21T04:06:21.156 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:21.168 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-21T04:06:21.229 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:21.229 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:21.230 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image3 2023-10-21T04:06:21.276 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-21T04:06:22.165 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:22.182 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-21T04:06:22.243 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:22.244 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:22.244 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image4 2023-10-21T04:06:22.294 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-21T04:06:23.169 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:23.182 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-21T04:06:23.244 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:23.244 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:23.244 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image5 2023-10-21T04:06:23.294 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-21T04:06:24.183 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:24.199 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-21T04:06:24.260 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:24.260 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:24.260 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image6 2023-10-21T04:06:24.310 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-21T04:06:25.183 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:25.196 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-21T04:06:25.260 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:25.261 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:25.261 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image7 2023-10-21T04:06:25.308 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-21T04:06:26.187 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:26.201 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-21T04:06:26.262 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:26.262 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:26.262 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image8 2023-10-21T04:06:26.311 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-21T04:06:27.206 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:27.218 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-21T04:06:27.279 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:27.279 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:27.279 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image9 2023-10-21T04:06:27.329 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-21T04:06:28.205 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:28.218 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-21T04:06:28.279 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:28.279 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:28.280 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image10 2023-10-21T04:06:28.329 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-21T04:06:29.222 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:29.236 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-21T04:06:29.299 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:29.299 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:29.299 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image11 2023-10-21T04:06:29.348 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-21T04:06:30.228 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:30.241 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-21T04:06:30.301 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:30.301 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:30.301 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image12 2023-10-21T04:06:30.350 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-21T04:06:31.227 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:31.241 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-21T04:06:31.302 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:31.302 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:31.302 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image13 2023-10-21T04:06:31.352 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-21T04:06:32.242 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:32.257 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-21T04:06:32.319 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:32.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:32.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image14 2023-10-21T04:06:32.369 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-21T04:06:33.243 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:33.256 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-21T04:06:33.316 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:33.317 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:33.317 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image15 2023-10-21T04:06:33.364 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-21T04:06:34.249 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:34.263 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-21T04:06:34.327 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:34.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:34.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image16 2023-10-21T04:06:34.375 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-21T04:06:35.262 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:35.276 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-21T04:06:35.338 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:35.338 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:35.338 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image17 2023-10-21T04:06:35.598 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-21T04:06:36.275 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:36.290 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-21T04:06:36.353 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:36.353 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:36.354 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image18 2023-10-21T04:06:36.403 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-21T04:06:37.281 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:37.295 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-21T04:06:37.357 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:37.358 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:37.358 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image19 2023-10-21T04:06:37.406 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-21T04:06:38.297 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:38.311 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-21T04:06:38.374 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:38.374 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:38.375 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image20 2023-10-21T04:06:38.424 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-21T04:06:39.298 INFO:tasks.workunit.client.0.smithi119.stdout:Mirroring enabled 2023-10-21T04:06:39.309 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-21T04:06:39.367 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:39.368 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:39.368 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i = 1 )) 2023-10-21T04:06:39.368 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:39.369 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image1 2023-10-21T04:06:39.492 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-21T04:06:39.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:39.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:39.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:39.493 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image2 2023-10-21T04:06:39.563 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-21T04:06:39.564 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:39.564 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:39.564 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:39.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image3 2023-10-21T04:06:39.637 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-21T04:06:39.637 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:39.637 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:39.637 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:39.638 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image4 2023-10-21T04:06:39.713 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-21T04:06:39.713 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:39.714 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:39.714 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:39.715 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image5 2023-10-21T04:06:39.784 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-21T04:06:39.784 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:39.785 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:39.785 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:39.785 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image6 2023-10-21T04:06:39.868 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-21T04:06:39.868 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:39.869 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:39.869 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:39.869 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image7 2023-10-21T04:06:39.966 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-21T04:06:39.967 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:39.967 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:39.967 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image8 2023-10-21T04:06:39.967 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:40.089 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-21T04:06:40.090 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:40.090 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:40.090 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:40.090 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image9 2023-10-21T04:06:40.209 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-21T04:06:40.209 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:40.210 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:40.210 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:40.211 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image10 2023-10-21T04:06:40.347 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-21T04:06:40.348 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:40.348 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:40.348 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image11 2023-10-21T04:06:40.349 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:40.460 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-21T04:06:40.460 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:40.460 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:40.461 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:40.461 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image12 2023-10-21T04:06:40.575 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-21T04:06:40.576 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:40.576 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:40.576 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:40.576 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image13 2023-10-21T04:06:40.742 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-21T04:06:40.742 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:40.742 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:40.742 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:40.743 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image14 2023-10-21T04:06:40.873 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-21T04:06:40.874 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:40.874 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:40.874 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image15 2023-10-21T04:06:40.874 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:41.019 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-21T04:06:41.020 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:41.020 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:41.020 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:41.023 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image16 2023-10-21T04:06:41.167 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-21T04:06:41.167 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:41.167 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:41.167 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:41.168 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image17 2023-10-21T04:06:41.329 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-21T04:06:41.330 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:41.330 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:41.330 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:41.331 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image18 2023-10-21T04:06:41.514 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-21T04:06:41.515 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:41.515 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:41.515 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:41.516 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image19 2023-10-21T04:06:41.631 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-21T04:06:41.631 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:41.632 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:41.632 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:41.632 INFO:tasks.workunit.client.0.smithi119.stderr:++ sudo rbd device map rbd/image20 2023-10-21T04:06:41.808 INFO:tasks.workunit.client.0.smithi119.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-21T04:06:41.808 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T04:06:41.808 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 20 )) 2023-10-21T04:06:41.809 INFO:tasks.workunit.client.0.smithi119.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:06:41.809 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:06:41.809 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861201 2023-10-21T04:06:41.810 INFO:tasks.workunit.client.0.smithi119.stderr:+ END_TIME=1697864801 2023-10-21T04:06:41.810 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR= 2023-10-21T04:06:41.810 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR= 2023-10-21T04:06:41.810 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:06:41.810 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n '' ]] 2023-10-21T04:06:41.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:06:50.907 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:06:50.900+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T04:06:50.907 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:06:50.900+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130c30 2023-10-21T04:06:50.908 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:06:50.900+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:06:50.908 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:06:50.900+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:06:50.908 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:06:50.900+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:06:51.810 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:06:51.811 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:06:51.811 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:06:51.825 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:06:52.683 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2444104118 2023-10-21T04:06:52.683 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:06:52.684 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861212 2023-10-21T04:06:52.684 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:06:52.684 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2444104118 ]] 2023-10-21T04:06:52.684 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2444104118 != '' ]] 2023-10-21T04:06:52.684 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/2444104118 2023-10-21T04:06:55.072 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/2444104118 until 2023-10-21T05:06:54.265420+0000 (3600 sec) 2023-10-21T04:06:55.121 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:06:55.125 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/2444104118 2023-10-21T04:06:55.820 INFO:tasks.workunit.client.0.smithi119.stderr:listed 1 entries 2023-10-21T04:06:55.846 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/2444104118 2023-10-21T04:06:55.847 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:07:00.009 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.000+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.010 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.000+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625df98ac00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.010 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.004+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.010 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.004+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625df98b280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.013 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.008+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.013 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.008+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625ddf69c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.023 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.020+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.023 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.020+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e04bbe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.035 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.032+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.035 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.032+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625ddf69c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.043 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.040+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.043 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.040+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625df98b300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.047 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.044+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.047 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.044+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625df98a380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.056 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.052+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.056 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.052+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0380a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.072 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.068+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.072 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.068+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625d8649580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.079 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.076+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.079 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.076+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0381a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.091 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.088+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.091 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.088+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625df98b780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.096 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.092+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.096 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.092+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625df98ac00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.109 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.108+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.109 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.108+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e04bbd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.118 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.116+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.118 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.116+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625ddf69c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.129 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.128+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.130 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.128+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625df98b100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.139 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.136+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.139 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.136+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625df98a100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.145 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.140+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.145 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.144+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625df98a380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.149 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.144+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.149 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.144+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0381a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.153 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.152+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.154 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.152+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625d8649580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.162 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.160+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:00.162 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:07:00.160+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625df98a900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:04.225 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:07:04.224+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:07:05.848 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:07:05.849 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:05.857 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:05.857 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:06.717 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2444104118 2023-10-21T04:07:06.717 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:07:06.718 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861226 2023-10-21T04:07:06.718 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:06.719 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2444104118 ]] 2023-10-21T04:07:06.719 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2444104118 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\4\4\4\1\0\4\1\1\8 ]] 2023-10-21T04:07:06.719 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:07:16.720 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:07:16.720 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:16.733 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:16.736 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:17.610 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2444104118 2023-10-21T04:07:17.610 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:07:17.611 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861237 2023-10-21T04:07:17.611 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:17.611 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2444104118 ]] 2023-10-21T04:07:17.612 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2444104118 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\4\4\4\1\0\4\1\1\8 ]] 2023-10-21T04:07:17.612 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:07:27.615 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:27.615 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:07:27.616 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:27.639 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:28.576 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2444104118 2023-10-21T04:07:28.576 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:07:28.578 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861248 2023-10-21T04:07:28.578 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:28.578 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2444104118 ]] 2023-10-21T04:07:28.579 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2444104118 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\4\4\4\1\0\4\1\1\8 ]] 2023-10-21T04:07:28.579 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:07:38.578 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:38.579 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:38.579 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:07:38.582 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:39.425 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2444104118 2023-10-21T04:07:39.425 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:07:39.426 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861259 2023-10-21T04:07:39.426 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:39.426 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2444104118 ]] 2023-10-21T04:07:39.427 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2444104118 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\4\4\4\1\0\4\1\1\8 ]] 2023-10-21T04:07:39.427 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:07:49.428 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:49.428 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:49.429 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:49.436 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:07:49.536 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:07:49.531+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T04:07:49.536 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:07:49.531+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130c30 2023-10-21T04:07:49.537 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:07:49.531+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:07:49.537 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:07:49.531+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:07:49.537 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:07:49.531+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:07:50.301 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2444104118 2023-10-21T04:07:50.301 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:07:50.301 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861270 2023-10-21T04:07:50.302 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:50.302 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2444104118 ]] 2023-10-21T04:07:50.302 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2444104118 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\4\4\4\1\0\4\1\1\8 ]] 2023-10-21T04:07:50.302 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:08:00.305 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:08:00.305 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:00.325 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:00.332 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:01.210 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1247365587 2023-10-21T04:08:01.211 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:08:01.211 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861281 2023-10-21T04:08:01.211 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:01.211 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1247365587 ]] 2023-10-21T04:08:01.211 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1247365587 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\4\4\4\1\0\4\1\1\8 ]] 2023-10-21T04:08:01.212 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/1247365587 2023-10-21T04:08:02.121 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::ImageWatcher: 0x5625e081f200 image watch failed: 94720673974272, (107) Transport endpoint is not connected 2023-10-21T04:08:02.121 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e081f200 handle_error: handle=94720673974272: (107) Transport endpoint is not connected 2023-10-21T04:08:02.122 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::ImageWatcher: 0x5625e09f0900 image watch failed: 94720682175488, (107) Transport endpoint is not connected 2023-10-21T04:08:02.122 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e09f0900 handle_error: handle=94720682175488: (107) Transport endpoint is not connected 2023-10-21T04:08:02.122 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::ImageWatcher: 0x5625e081e600 image watch failed: 94720676825088, (107) Transport endpoint is not connected 2023-10-21T04:08:02.122 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e081e600 handle_error: handle=94720676825088: (107) Transport endpoint is not connected 2023-10-21T04:08:02.123 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::watcher::RewatchRequest: 0x5625db49e050 handle_unwatch client blocklisted 2023-10-21T04:08:02.123 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e081e600 handle_rewatch: client blocklisted 2023-10-21T04:08:02.123 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::watcher::RewatchRequest: 0x5625e0186cd0 handle_unwatch client blocklisted 2023-10-21T04:08:02.124 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e081f200 handle_rewatch: client blocklisted 2023-10-21T04:08:02.124 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::watcher::RewatchRequest: 0x5625e09c8780 handle_unwatch client blocklisted 2023-10-21T04:08:02.124 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e09f0900 handle_rewatch: client blocklisted 2023-10-21T04:08:02.124 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::ImageWatcher: 0x5625e09f0c00 image watch failed: 94720666710016, (107) Transport endpoint is not connected 2023-10-21T04:08:02.124 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e09f0c00 handle_error: handle=94720666710016: (107) Transport endpoint is not connected 2023-10-21T04:08:02.125 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::ImageWatcher: 0x5625e04da000 image watch failed: 94720674491392, (107) Transport endpoint is not connected 2023-10-21T04:08:02.125 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e04da000 handle_error: handle=94720674491392: (107) Transport endpoint is not connected 2023-10-21T04:08:02.125 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::watcher::RewatchRequest: 0x5625e0391e00 handle_unwatch client blocklisted 2023-10-21T04:08:02.125 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e09f0c00 handle_rewatch: client blocklisted 2023-10-21T04:08:02.125 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::ManagedLock: 0x5625e0243a18 send_acquire_lock: watcher not registered - client blocklisted 2023-10-21T04:08:02.126 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625dfaf8750 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:02.126 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e098e6e0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:02.127 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb9d6640 -1 librbd::watcher::RewatchRequest: 0x5625e09be960 handle_unwatch client blocklisted 2023-10-21T04:08:02.127 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb9d6640 -1 librbd::Watcher: 0x5625e04da000 handle_rewatch: client blocklisted 2023-10-21T04:08:02.127 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb9d6640 -1 librbd::ManagedLock: 0x5625e01bba18 send_acquire_lock: watcher not registered - client blocklisted 2023-10-21T04:08:02.127 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb9d6640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625dfaf81a0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:02.128 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb9d6640 -1 librbd::io::AioCompletion: 0x5625e041bce0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:02.128 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb9d6640 -1 librbd::ImageWatcher: 0x5625e081ec00 image watch failed: 94720666711040, (107) Transport endpoint is not connected 2023-10-21T04:08:02.128 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb9d6640 -1 librbd::Watcher: 0x5625e081ec00 handle_error: handle=94720666711040: (107) Transport endpoint is not connected 2023-10-21T04:08:02.128 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb9d6640 -1 librbd::ImageWatcher: 0x5625e02d6000 image watch failed: 94720677446656, (107) Transport endpoint is not connected 2023-10-21T04:08:02.128 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb9d6640 -1 librbd::Watcher: 0x5625e02d6000 handle_error: handle=94720677446656: (107) Transport endpoint is not connected 2023-10-21T04:08:02.129 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb9d6640 -1 librbd::ImageWatcher: 0x5625e02d2600 image watch failed: 94720682174464, (107) Transport endpoint is not connected 2023-10-21T04:08:02.129 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.119+0000 7ffacb9d6640 -1 librbd::Watcher: 0x5625e02d2600 handle_error: handle=94720682174464: (107) Transport endpoint is not connected 2023-10-21T04:08:02.129 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.123+0000 7ffacb9d6640 -1 librbd::ImageWatcher: 0x5625e081e000 image watch failed: 94720677189632, (107) Transport endpoint is not connected 2023-10-21T04:08:02.129 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.123+0000 7ffacb9d6640 -1 librbd::Watcher: 0x5625e081e000 handle_error: handle=94720677189632: (107) Transport endpoint is not connected 2023-10-21T04:08:02.129 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.123+0000 7ffacb9d6640 -1 librbd::watcher::RewatchRequest: 0x5625e0447e00 handle_unwatch client blocklisted 2023-10-21T04:08:02.130 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.123+0000 7ffacb9d6640 -1 librbd::Watcher: 0x5625e081ec00 handle_rewatch: client blocklisted 2023-10-21T04:08:02.130 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.123+0000 7ffacb9d6640 -1 librbd::watcher::RewatchRequest: 0x5625e03ee9b0 handle_unwatch client blocklisted 2023-10-21T04:08:02.130 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.123+0000 7ffacb9d6640 -1 librbd::Watcher: 0x5625e02d6000 handle_rewatch: client blocklisted 2023-10-21T04:08:02.130 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.123+0000 7ffacb9d6640 -1 librbd::watcher::RewatchRequest: 0x5625e09c94f0 handle_unwatch client blocklisted 2023-10-21T04:08:02.131 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.123+0000 7ffacb9d6640 -1 librbd::Watcher: 0x5625e02d2600 handle_rewatch: client blocklisted 2023-10-21T04:08:02.131 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.123+0000 7ffacb9d6640 -1 librbd::watcher::RewatchRequest: 0x5625e03ee9b0 handle_unwatch client blocklisted 2023-10-21T04:08:02.131 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.123+0000 7ffacb9d6640 -1 librbd::Watcher: 0x5625e081e000 handle_rewatch: client blocklisted 2023-10-21T04:08:02.132 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.127+0000 7ffacb9d6640 -1 librbd::ImageWatcher: 0x5625e081e900 image watch failed: 94720666707968, (107) Transport endpoint is not connected 2023-10-21T04:08:02.132 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.127+0000 7ffacb9d6640 -1 librbd::Watcher: 0x5625e081e900 handle_error: handle=94720666707968: (107) Transport endpoint is not connected 2023-10-21T04:08:02.132 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.127+0000 7ffacb9d6640 -1 librbd::watcher::RewatchRequest: 0x5625e0123e00 handle_unwatch client blocklisted 2023-10-21T04:08:02.132 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:02.127+0000 7ffacb9d6640 -1 librbd::Watcher: 0x5625e081e900 handle_rewatch: client blocklisted 2023-10-21T04:08:03.113 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/1247365587 until 2023-10-21T05:08:02.299456+0000 (3600 sec) 2023-10-21T04:08:03.119 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.115+0000 7ffacb9d6640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.120 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.115+0000 7ffacb9d6640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625dfaf9ba0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.120 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.115+0000 7ffacb9d6640 -1 librbd::io::AioCompletion: 0x5625e07429a0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.120 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.115+0000 7ffacb1d5640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.121 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.115+0000 7ffacb1d5640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.121 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.115+0000 7ffacb1d5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625dfaf9c70 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.121 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.115+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e08762c0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.121 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.115+0000 7ffacb1d5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625dfaf95f0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.121 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.115+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e04ec580 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.122 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.123 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625e01988f0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.123 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e054f080 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.123 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.123 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.123 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.124 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.124 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625dfaf88f0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.124 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e098fe40 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.124 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625dfaf9790 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.124 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e098f4a0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.125 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625dfaf9860 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.125 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e098edc0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.125 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625dfaf9d40 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.125 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e0742420 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.125 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.126 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.126 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.126 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.126 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.126 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e04ec580 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.127 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5625e078e740 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.127 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e098e160 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.127 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e054f600 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.127 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e054f8c0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.127 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5625e08637c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.128 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5625e09b6700 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.128 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5625e09e8300 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.128 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e054fb80 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.128 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb1d5640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5625e09e80c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.129 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb9d6640 -1 librbd::managed_lock::ReleaseRequest: 0x5625e01d9b60 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.129 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb9d6640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.129 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb9d6640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.130 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb9d6640 -1 librbd::managed_lock::ReleaseRequest: 0x5625e07479e0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.130 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb9d6640 -1 librbd::managed_lock::ReleaseRequest: 0x5625e04e3380 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.130 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb9d6640 -1 librbd::io::AioCompletion: 0x5625e098e6e0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.130 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb9d6640 -1 librbd::io::AioCompletion: 0x5625e054f1e0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.131 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb9d6640 -1 librbd::managed_lock::ReleaseRequest: 0x5625e01855c0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.131 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb9d6640 -1 librbd::managed_lock::ReleaseRequest: 0x5625db4b7860 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.131 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb9d6640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5625e09e8600 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.131 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.119+0000 7ffacb9d6640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5625e09e82c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.133 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.123+0000 7ffacb1d5640 -1 librbd::managed_lock::ReleaseRequest: 0x5625e0184720 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.134 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.123+0000 7ffacb1d5640 -1 librbd::managed_lock::ReleaseRequest: 0x5625e07479e0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.134 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.123+0000 7ffacb1d5640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.134 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.123+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e041bce0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.134 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.123+0000 7ffacb1d5640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5625e0918400 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.135 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:08:03.123+0000 7ffacb9d6640 -1 librbd::managed_lock::ReleaseRequest: 0x5625e07478c0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:08:03.141 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:08:03.142 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/1247365587 2023-10-21T04:08:03.800 INFO:tasks.workunit.client.0.smithi119.stderr:listed 2 entries 2023-10-21T04:08:03.826 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/1247365587 2023-10-21T04:08:03.826 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:08:13.830 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:08:13.830 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:13.831 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:13.848 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:14.772 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1247365587 2023-10-21T04:08:14.773 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:08:14.773 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861294 2023-10-21T04:08:14.773 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:14.774 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1247365587 ]] 2023-10-21T04:08:14.774 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1247365587 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\4\7\3\6\5\5\8\7 ]] 2023-10-21T04:08:14.774 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:08:24.775 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:24.776 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:08:24.776 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:24.777 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:25.849 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1247365587 2023-10-21T04:08:25.849 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:08:25.851 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861305 2023-10-21T04:08:25.851 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:25.851 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1247365587 ]] 2023-10-21T04:08:25.852 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1247365587 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\4\7\3\6\5\5\8\7 ]] 2023-10-21T04:08:25.852 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:08:35.851 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:08:35.853 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:35.853 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:35.854 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:36.829 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1247365587 2023-10-21T04:08:36.833 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:08:36.835 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861316 2023-10-21T04:08:36.835 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:36.835 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1247365587 ]] 2023-10-21T04:08:36.835 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1247365587 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\4\7\3\6\5\5\8\7 ]] 2023-10-21T04:08:36.835 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:08:38.779 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:08:38.775+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:08:38.779 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:08:38.775+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:08:38.780 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:08:38.775+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:08:38.780 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:08:38.775+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T04:08:38.780 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:08:38.775+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:08:38.780 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:08:38.775+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:08:38.780 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:08:38.775+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:08:38.781 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:08:38.775+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:08:38.781 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:08:38.775+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:08:38.781 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:08:38.775+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:08:46.836 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:08:46.837 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:46.838 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:46.859 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:47.832 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1247365587 2023-10-21T04:08:47.833 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:08:47.833 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861327 2023-10-21T04:08:47.833 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:47.834 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1247365587 ]] 2023-10-21T04:08:47.834 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1247365587 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\4\7\3\6\5\5\8\7 ]] 2023-10-21T04:08:47.834 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:08:57.841 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:08:57.845 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:57.849 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:57.850 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:58.759 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1247365587 2023-10-21T04:08:58.759 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:08:58.760 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861338 2023-10-21T04:08:58.760 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:58.760 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1247365587 ]] 2023-10-21T04:08:58.760 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1247365587 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\4\7\3\6\5\5\8\7 ]] 2023-10-21T04:08:58.761 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:09:08.761 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:08.762 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:08.768 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:08.771 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:09:09.652 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2787404687 2023-10-21T04:09:09.652 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:09:09.652 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861349 2023-10-21T04:09:09.653 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:09.653 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2787404687 ]] 2023-10-21T04:09:09.653 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2787404687 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\4\7\3\6\5\5\8\7 ]] 2023-10-21T04:09:09.653 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/2787404687 2023-10-21T04:09:11.134 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/2787404687 until 2023-10-21T05:09:10.339120+0000 (3600 sec) 2023-10-21T04:09:11.169 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:09:11.169 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/2787404687 2023-10-21T04:09:11.821 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:09:11.822+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:09:11.822 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:09:11.822+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:09:11.822 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:09:11.822+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:09:11.822 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:09:11.822+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:09:11.822 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:09:11.822+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:09:11.823 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:09:11.822+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:09:11.823 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:09:11.822+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:09:11.823 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:09:11.822+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:09:11.823 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:09:11.822+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:09:11.875 INFO:tasks.workunit.client.0.smithi119.stderr:listed 3 entries 2023-10-21T04:09:11.910 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/2787404687 2023-10-21T04:09:11.911 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:09:21.912 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:21.913 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:21.913 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:09:21.924 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:22.819 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2787404687 2023-10-21T04:09:22.820 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:09:22.821 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861362 2023-10-21T04:09:22.822 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:22.822 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2787404687 ]] 2023-10-21T04:09:22.822 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2787404687 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\8\7\4\0\4\6\8\7 ]] 2023-10-21T04:09:22.822 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:09:32.834 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:09:32.834 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:32.834 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:32.836 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:33.726 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2787404687 2023-10-21T04:09:33.726 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:09:33.727 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861373 2023-10-21T04:09:33.727 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:33.727 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2787404687 ]] 2023-10-21T04:09:33.728 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2787404687 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\8\7\4\0\4\6\8\7 ]] 2023-10-21T04:09:33.728 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:09:43.729 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:09:43.729 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:43.730 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:43.730 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:44.834 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2787404687 2023-10-21T04:09:44.835 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:09:44.840 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861384 2023-10-21T04:09:44.840 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:44.840 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2787404687 ]] 2023-10-21T04:09:44.840 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2787404687 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\8\7\4\0\4\6\8\7 ]] 2023-10-21T04:09:44.841 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:09:54.849 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:09:54.849 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:54.857 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:54.859 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:55.801 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2787404687 2023-10-21T04:09:55.801 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:09:55.802 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861395 2023-10-21T04:09:55.802 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:55.802 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2787404687 ]] 2023-10-21T04:09:55.803 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2787404687 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\8\7\4\0\4\6\8\7 ]] 2023-10-21T04:09:55.803 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:10:00.001 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.001+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.001 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.001+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db4b3080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.004 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.005+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.004 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.005+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db4b3280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.006 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.005+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.006 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.005+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db4b3480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.008 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.009+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.009 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.009+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db4b3680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.011 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.009+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.011 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.009+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db4b3880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.018 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.017+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.018 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.017+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.018 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.017+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db4b3a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.018 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.017+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db4b3c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.019 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.017+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.019 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.017+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db4b3e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.022 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.021+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.022 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.021+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625dfcdc080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.025 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.025+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.025 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.025+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625dfcdc280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.031 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.029+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.031 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.029+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625db4b3100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.034 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.033+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.034 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.033+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0420400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.039 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.037+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.039 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.037+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625db4b3680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.043 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.041+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.043 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.041+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625db4b3400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.049 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.049+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.050 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.049+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625db4b3200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.055 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.053+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.055 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.053+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e06c9300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.061 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.061+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.061 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.061+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625dfcdc600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.079 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.077+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.080 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.077+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625db4b3800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.087 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.085+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.087 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.085+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625db4b3a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.098 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.097+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.098 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:10:00.097+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625dfcdc700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:05.803 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:05.804 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:10:05.804 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:05.804 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:06.761 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2787404687 2023-10-21T04:10:06.762 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:10:06.762 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861406 2023-10-21T04:10:06.763 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:06.763 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2787404687 ]] 2023-10-21T04:10:06.763 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2787404687 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\8\7\4\0\4\6\8\7 ]] 2023-10-21T04:10:06.763 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:10:16.769 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:10:16.769 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:16.773 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:16.783 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:17.774 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2787404687 2023-10-21T04:10:17.775 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:10:17.775 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861417 2023-10-21T04:10:17.775 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:17.776 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2787404687 ]] 2023-10-21T04:10:17.776 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2787404687 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\8\7\4\0\4\6\8\7 ]] 2023-10-21T04:10:17.776 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:10:27.777 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:10:27.777 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:27.783 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:27.783 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:28.605 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1503047554 2023-10-21T04:10:28.606 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:10:28.606 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861428 2023-10-21T04:10:28.607 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:28.607 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1503047554 ]] 2023-10-21T04:10:28.607 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1503047554 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\8\7\4\0\4\6\8\7 ]] 2023-10-21T04:10:28.609 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/1503047554 2023-10-21T04:10:31.110 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/1503047554 until 2023-10-21T05:10:30.324437+0000 (3600 sec) 2023-10-21T04:10:31.148 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:10:31.148 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/1503047554 2023-10-21T04:10:31.824 INFO:tasks.workunit.client.0.smithi119.stderr:listed 4 entries 2023-10-21T04:10:31.849 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/1503047554 2023-10-21T04:10:31.849 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:10:41.851 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:41.855 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:41.855 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:10:41.856 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:42.711 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1503047554 2023-10-21T04:10:42.711 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:10:42.712 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861442 2023-10-21T04:10:42.712 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:42.712 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1503047554 ]] 2023-10-21T04:10:42.712 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1503047554 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\5\0\3\0\4\7\5\5\4 ]] 2023-10-21T04:10:42.712 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:10:52.713 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:10:52.715 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:52.716 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:52.729 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:53.579 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1503047554 2023-10-21T04:10:53.580 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:10:53.581 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861453 2023-10-21T04:10:53.581 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:53.581 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1503047554 ]] 2023-10-21T04:10:53.581 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1503047554 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\5\0\3\0\4\7\5\5\4 ]] 2023-10-21T04:10:53.581 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:10:58.824 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:10:58.825+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:10:58.824 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:10:58.825+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:11:00.001 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.001+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.002 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.001+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e08b0a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.004 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.005+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.004 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.005+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e08b0c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.009 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.009+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.010 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.009+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e08b0e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.019 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.017+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.019 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.017+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e08b1000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.029 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.029+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.029 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.029+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e08b1200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.033 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.033+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.033 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.033+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e08b1400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.038 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.037+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.038 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.037+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e08b1a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.043 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.041+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.044 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.041+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e08b1880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.046 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.045+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.047 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.045+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e08b1600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.050 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.049+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.050 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.049+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e08b1c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.059 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.057+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.059 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.057+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e08b1e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.068 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.069+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.068 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.069+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0740000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.076 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.077+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.076 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.077+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0740200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.079 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.081+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.079 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.081+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0740400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.083 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.085+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.084 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.085+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0740600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.094 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.093+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.095 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.093+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0740800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.098 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.097+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.099 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.097+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0740a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.100 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.101+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.100 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.101+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0740c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.104 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.105+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.104 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.105+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0740e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.109 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.109+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.109 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:11:00.109+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0741000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:03.582 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:03.583 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:03.587 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:03.600 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:11:04.589 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1503047554 2023-10-21T04:11:04.589 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:11:04.590 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861464 2023-10-21T04:11:04.590 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:04.590 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1503047554 ]] 2023-10-21T04:11:04.590 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1503047554 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\5\0\3\0\4\7\5\5\4 ]] 2023-10-21T04:11:04.590 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:11:14.591 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:14.593 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:11:14.594 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:14.594 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:15.620 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1503047554 2023-10-21T04:11:15.627 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:11:15.631 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861475 2023-10-21T04:11:15.631 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:15.632 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1503047554 ]] 2023-10-21T04:11:15.632 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1503047554 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\5\0\3\0\4\7\5\5\4 ]] 2023-10-21T04:11:15.632 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:11:25.641 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:11:25.641 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:25.646 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:25.655 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:26.541 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1503047554 2023-10-21T04:11:26.541 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:11:26.542 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861486 2023-10-21T04:11:26.542 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:26.542 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1503047554 ]] 2023-10-21T04:11:26.542 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1503047554 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\5\0\3\0\4\7\5\5\4 ]] 2023-10-21T04:11:26.543 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:11:36.543 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:11:36.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:36.550 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:36.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:37.456 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2943851828 2023-10-21T04:11:37.456 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:11:37.457 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861497 2023-10-21T04:11:37.457 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:37.457 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2943851828 ]] 2023-10-21T04:11:37.457 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2943851828 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\5\0\3\0\4\7\5\5\4 ]] 2023-10-21T04:11:37.458 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/2943851828 2023-10-21T04:11:39.153 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/2943851828 until 2023-10-21T05:11:38.408367+0000 (3600 sec) 2023-10-21T04:11:39.193 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:11:39.210 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/2943851828 2023-10-21T04:11:39.873 INFO:tasks.workunit.client.0.smithi119.stderr:listed 5 entries 2023-10-21T04:11:39.901 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/2943851828 2023-10-21T04:11:39.901 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:11:49.902 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:49.903 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:49.903 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:49.926 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:11:50.739 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2943851828 2023-10-21T04:11:50.739 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:11:50.740 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861510 2023-10-21T04:11:50.740 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:50.740 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2943851828 ]] 2023-10-21T04:11:50.741 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2943851828 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\9\4\3\8\5\1\8\2\8 ]] 2023-10-21T04:11:50.741 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:12:00.003 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.004+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.003 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.004+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db496d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.012 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.012+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.013 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.012+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db497800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.018 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.016+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.018 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.016+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db497980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.024 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.024+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.025 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.024+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db496500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.034 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.032+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.034 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.032+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0741280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.034 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.032+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.034 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.032+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0741c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.037 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.036+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.037 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.036+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0741e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.041 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.040+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.041 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.040+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e04ddd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.045 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.044+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.045 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.044+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c06000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.049 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.048+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.049 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.048+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c06200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.054 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.056+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.055 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.056+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c06400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.058 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.060+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.058 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.060+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c06600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.065 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.064+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.065 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.064+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c06800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.071 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.072+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.072 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.072+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c06a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.078 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.076+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.078 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.076+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c06c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.084 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.084+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.084 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.084+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c07400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.089 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.088+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.090 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.088+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c07280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.095 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.096+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.095 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.096+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c07080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.101 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.100+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.101 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.100+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c07b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.107 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.108+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.107 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:00.108+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c07a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.741 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:12:00.742 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:00.742 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:00.743 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:01.683 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2943851828 2023-10-21T04:12:01.684 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:12:01.685 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861521 2023-10-21T04:12:01.685 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:01.685 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2943851828 ]] 2023-10-21T04:12:01.685 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2943851828 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\9\4\3\8\5\1\8\2\8 ]] 2023-10-21T04:12:01.685 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:12:11.688 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:11.689 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:12:11.691 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:11.702 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:12.574 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2943851828 2023-10-21T04:12:12.574 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:12:12.590 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861532 2023-10-21T04:12:12.590 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:12.590 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2943851828 ]] 2023-10-21T04:12:12.590 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2943851828 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\9\4\3\8\5\1\8\2\8 ]] 2023-10-21T04:12:12.591 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:12:22.591 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:22.592 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:22.596 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:12:22.606 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:23.450 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2943851828 2023-10-21T04:12:23.450 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:12:23.451 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861543 2023-10-21T04:12:23.451 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:23.452 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2943851828 ]] 2023-10-21T04:12:23.452 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2943851828 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\9\4\3\8\5\1\8\2\8 ]] 2023-10-21T04:12:23.452 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:12:33.452 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:33.453 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:33.454 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:33.474 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:12:34.447 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2943851828 2023-10-21T04:12:34.448 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:12:34.448 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861554 2023-10-21T04:12:34.449 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:34.449 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2943851828 ]] 2023-10-21T04:12:34.449 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2943851828 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\9\4\3\8\5\1\8\2\8 ]] 2023-10-21T04:12:34.449 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:12:44.449 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:12:44.450 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:44.450 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:44.451 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:45.306 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2732595255 2023-10-21T04:12:45.306 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:12:45.307 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861565 2023-10-21T04:12:45.307 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:45.307 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2732595255 ]] 2023-10-21T04:12:45.308 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2732595255 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\9\4\3\8\5\1\8\2\8 ]] 2023-10-21T04:12:45.308 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/2732595255 2023-10-21T04:12:47.148 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/2732595255 until 2023-10-21T05:12:46.328348+0000 (3600 sec) 2023-10-21T04:12:47.188 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:12:47.194 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/2732595255 2023-10-21T04:12:47.990 INFO:tasks.workunit.client.0.smithi119.stderr:listed 6 entries 2023-10-21T04:12:48.015 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/2732595255 2023-10-21T04:12:48.016 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:12:58.021 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:58.022 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:58.029 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:58.030 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:12:59.027 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2732595255 2023-10-21T04:12:59.027 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:12:59.028 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861579 2023-10-21T04:12:59.028 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:59.028 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2732595255 ]] 2023-10-21T04:12:59.029 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2732595255 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\3\2\5\9\5\2\5\5 ]] 2023-10-21T04:12:59.029 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:13:00.002 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:59.999+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.002 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:12:59.999+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625df98b880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.003 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.003+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.003 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.003+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625df90c280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.006 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.007+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.006 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.007+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625db496680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.009 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.011+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.010 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.011+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625db496e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.013 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.011+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.013 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.011+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c07c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.017 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.019+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.018 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.019+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c07800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.024 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.023+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.025 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.023+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c07d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.031 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.031+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.031 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.031+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0c07f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.035 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.035+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.035 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.035+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0c4e100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.039 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.039+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.040 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.039+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0c4e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.043 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.043+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.043 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.043+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0c4e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.047 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.047+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.047 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.047+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0c4e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.051 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.051+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.051 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.051+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0c4e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.059 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.059+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.059 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.059+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.060 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.059+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c4eb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.060 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.059+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c4ed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.060 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.059+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.060 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.059+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c4ef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.067 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.067+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.067 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.067+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db497980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.070 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.071+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.070 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.071+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625db496a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.073 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.075+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.074 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.075+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0740a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.077 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.075+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.078 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:00.075+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0380600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:02.726 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:13:02.727+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T04:13:02.727 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:13:02.727+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130c30 2023-10-21T04:13:02.727 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:13:02.727+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:13:02.727 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:13:02.727+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:13:02.727 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:13:02.727+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:13:09.048 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:13:09.053 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:09.053 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:09.072 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:09.914 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2732595255 2023-10-21T04:13:09.920 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:13:09.921 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861589 2023-10-21T04:13:09.922 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:09.922 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2732595255 ]] 2023-10-21T04:13:09.922 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2732595255 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\3\2\5\9\5\2\5\5 ]] 2023-10-21T04:13:09.922 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:13:19.929 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:19.929 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:19.933 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:13:19.940 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:20.919 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2732595255 2023-10-21T04:13:20.920 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:13:20.921 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861600 2023-10-21T04:13:20.921 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:20.921 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2732595255 ]] 2023-10-21T04:13:20.921 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2732595255 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\3\2\5\9\5\2\5\5 ]] 2023-10-21T04:13:20.922 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:13:30.922 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:13:30.923 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:30.937 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:30.937 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:31.872 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2732595255 2023-10-21T04:13:31.872 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:13:31.874 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861611 2023-10-21T04:13:31.874 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:31.875 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2732595255 ]] 2023-10-21T04:13:31.875 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2732595255 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\3\2\5\9\5\2\5\5 ]] 2023-10-21T04:13:31.875 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:13:31.973 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:13:31.971+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:13:31.973 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:13:31.971+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:13:41.880 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:13:41.880 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:41.881 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:41.881 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:42.750 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2732595255 2023-10-21T04:13:42.751 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:13:42.752 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861622 2023-10-21T04:13:42.752 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:42.752 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2732595255 ]] 2023-10-21T04:13:42.752 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2732595255 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\3\2\5\9\5\2\5\5 ]] 2023-10-21T04:13:42.752 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:13:52.753 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:52.754 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:52.757 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:13:52.778 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:53.721 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1819191538 2023-10-21T04:13:53.722 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:13:53.722 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861633 2023-10-21T04:13:53.723 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:53.723 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1819191538 ]] 2023-10-21T04:13:53.723 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1819191538 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\3\2\5\9\5\2\5\5 ]] 2023-10-21T04:13:53.723 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/1819191538 2023-10-21T04:13:56.171 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/1819191538 until 2023-10-21T05:13:55.512958+0000 (3600 sec) 2023-10-21T04:13:56.205 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:13:56.206 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/1819191538 2023-10-21T04:13:56.954 INFO:tasks.workunit.client.0.smithi119.stderr:listed 7 entries 2023-10-21T04:13:56.983 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/1819191538 2023-10-21T04:13:56.983 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:14:00.000 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:59.998+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.000 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:59.998+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625dfcdda00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.000 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:59.998+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.001 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:13:59.998+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625db497380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.008 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.006+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.008 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.006+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0c4e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.010 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.010+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.010 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.010+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0c4f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.018 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.018+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.018 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.018+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0c4f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.034 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.034+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.034 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.034+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0c4f480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.039 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.038+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.040 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.038+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0c4e000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.045 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.046+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.045 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.046+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0c4fb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.052 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.050+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.052 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.050+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0740f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.057 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.058+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.057 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.058+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0740b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.062 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.062+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.062 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.062+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0c4fa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.067 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.066+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.067 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.066+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0c4e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.073 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.074+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.073 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.074+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625df98ab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.079 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.078+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.079 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.078+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0740080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.090 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.090+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.090 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.090+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0d0c180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.094 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.094+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.094 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.094+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0d0c380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.099 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.098+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.099 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.098+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0d0c580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.105 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.102+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.105 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.102+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0d0c780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.111 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.110+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.111 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.110+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0d0c980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.118 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.114+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:00.118 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:14:00.114+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0d0cb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:06.985 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:06.986 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:06.986 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:06.996 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:14:07.937 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1819191538 2023-10-21T04:14:07.938 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:14:07.938 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861647 2023-10-21T04:14:07.938 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:07.939 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1819191538 ]] 2023-10-21T04:14:07.939 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1819191538 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\8\1\9\1\9\1\5\3\8 ]] 2023-10-21T04:14:07.939 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:14:17.940 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:17.946 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:14:17.946 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:17.948 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:18.951 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1819191538 2023-10-21T04:14:18.951 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:14:18.952 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861658 2023-10-21T04:14:18.952 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:18.952 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1819191538 ]] 2023-10-21T04:14:18.953 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1819191538 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\8\1\9\1\9\1\5\3\8 ]] 2023-10-21T04:14:18.953 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:14:26.971 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:14:26.974+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:14:26.971 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:14:26.974+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:14:26.972 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:14:26.974+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:14:26.972 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:14:26.974+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T04:14:26.972 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:14:26.974+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:14:26.972 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:14:26.974+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:14:26.973 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:14:26.974+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:14:26.973 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:14:26.974+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:14:26.973 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:14:26.974+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:14:26.973 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:14:26.974+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:14:28.955 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:14:28.956 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:28.956 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:28.960 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:29.922 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1819191538 2023-10-21T04:14:29.922 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:14:29.923 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861669 2023-10-21T04:14:29.923 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:29.923 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1819191538 ]] 2023-10-21T04:14:29.923 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1819191538 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\8\1\9\1\9\1\5\3\8 ]] 2023-10-21T04:14:29.923 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:14:39.925 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:39.925 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:39.927 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:39.929 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:14:40.809 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1819191538 2023-10-21T04:14:40.810 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:14:40.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861680 2023-10-21T04:14:40.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:40.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1819191538 ]] 2023-10-21T04:14:40.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1819191538 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\8\1\9\1\9\1\5\3\8 ]] 2023-10-21T04:14:40.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:14:50.812 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:50.812 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:14:50.813 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:50.819 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:51.625 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1819191538 2023-10-21T04:14:51.625 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:14:51.626 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861691 2023-10-21T04:14:51.626 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:51.626 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1819191538 ]] 2023-10-21T04:14:51.626 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1819191538 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\8\1\9\1\9\1\5\3\8 ]] 2023-10-21T04:14:51.627 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:15:01.632 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:01.632 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:01.635 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:01.636 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:15:02.460 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/4093318189 2023-10-21T04:15:02.460 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:15:02.461 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861702 2023-10-21T04:15:02.461 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:02.462 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/4093318189 ]] 2023-10-21T04:15:02.462 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/4093318189 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\8\1\9\1\9\1\5\3\8 ]] 2023-10-21T04:15:02.462 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/4093318189 2023-10-21T04:15:03.234 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.238+0000 7ffacb1d5640 -1 librbd::ImageWatcher: 0x5625e0dc6600 image watch failed: 94720687835136, (107) Transport endpoint is not connected 2023-10-21T04:15:03.235 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.238+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e0dc6600 handle_error: handle=94720687835136: (107) Transport endpoint is not connected 2023-10-21T04:15:03.235 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.238+0000 7ffacb1d5640 -1 librbd::watcher::RewatchRequest: 0x5625e0885950 handle_unwatch client blocklisted 2023-10-21T04:15:03.235 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.238+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e0dc6600 handle_rewatch: client blocklisted 2023-10-21T04:15:03.308 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.310+0000 7ffacb1d5640 -1 librbd::ImageWatcher: 0x5625e0dc7500 image watch failed: 94720687680512, (107) Transport endpoint is not connected 2023-10-21T04:15:03.308 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.310+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e0dc7500 handle_error: handle=94720687680512: (107) Transport endpoint is not connected 2023-10-21T04:15:03.308 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.310+0000 7ffacb1d5640 -1 librbd::ImageWatcher: 0x5625e0dc7200 image watch failed: 94720686333952, (107) Transport endpoint is not connected 2023-10-21T04:15:03.308 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.310+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e0dc7200 handle_error: handle=94720686333952: (107) Transport endpoint is not connected 2023-10-21T04:15:03.309 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.314+0000 7ffacb1d5640 -1 librbd::image::RefreshRequest: failed to retrieve snapshots: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:03.309 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.314+0000 7ffacb1d5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625e01980d0 handle_refresh_image: failed to refresh image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:03.309 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.314+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e0f0dce0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:03.368 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.370+0000 7ffacb1d5640 -1 librbd::watcher::RewatchRequest: 0x5625e046bb80 handle_unwatch client blocklisted 2023-10-21T04:15:03.368 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.370+0000 7ffacb1d5640 -1 librbd::watcher::RewatchRequest: 0x5625e046bb30 handle_unwatch client blocklisted 2023-10-21T04:15:03.368 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:03.370+0000 7ffacb1d5640 -1 librbd::Watcher: 0x5625e0dc7200 handle_rewatch: client blocklisted 2023-10-21T04:15:04.220 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/4093318189 until 2023-10-21T05:15:03.436061+0000 (3600 sec) 2023-10-21T04:15:04.235 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.238+0000 7ffacb1d5640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.235 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.238+0000 7ffacb1d5640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625e0198000 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.235 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.238+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e0e0c420 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.235 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.238+0000 7ffacb1d5640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.236 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.238+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e0f0dce0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.236 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.238+0000 7ffacb1d5640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5625e0f2bc00 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.236 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.238+0000 7ffacb1d5640 -1 librbd::managed_lock::ReleaseRequest: 0x5625e0b9c600 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.250 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:15:04.250 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/4093318189 2023-10-21T04:15:04.497 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.502+0000 7ffacb9d6640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.497 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.502+0000 7ffacb9d6640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5625e01992b0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.497 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.502+0000 7ffacb9d6640 -1 librbd::io::AioCompletion: 0x5625e0e0cb00 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.498 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.502+0000 7ffacb1d5640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.498 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.502+0000 7ffacb1d5640 -1 librbd::io::AioCompletion: 0x5625e09fc840 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.499 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.502+0000 7ffacb9d6640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5625e0ec2ec0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.499 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:15:04.502+0000 7ffacb9d6640 -1 librbd::managed_lock::ReleaseRequest: 0x5625e01d96e0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:15:04.938 INFO:tasks.workunit.client.0.smithi119.stderr:listed 8 entries 2023-10-21T04:15:04.960 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/4093318189 2023-10-21T04:15:04.961 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:15:14.962 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:14.965 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:14.976 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:15:14.976 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:15.824 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/4093318189 2023-10-21T04:15:15.824 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:15:15.825 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861715 2023-10-21T04:15:15.825 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:15.825 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/4093318189 ]] 2023-10-21T04:15:15.826 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/4093318189 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\4\0\9\3\3\1\8\1\8\9 ]] 2023-10-21T04:15:15.826 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:15:25.826 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:15:25.827 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:25.827 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:25.849 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:26.700 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/4093318189 2023-10-21T04:15:26.700 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:15:26.701 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861726 2023-10-21T04:15:26.701 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:26.702 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/4093318189 ]] 2023-10-21T04:15:26.702 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/4093318189 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\4\0\9\3\3\1\8\1\8\9 ]] 2023-10-21T04:15:26.702 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:15:33.214 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:15:33.217+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:15:33.231 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:15:33.233+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:15:33.232 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:15:33.237+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:15:36.708 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:15:36.710 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:36.711 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:36.717 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:37.640 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/4093318189 2023-10-21T04:15:37.640 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:15:37.641 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861737 2023-10-21T04:15:37.641 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:37.641 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/4093318189 ]] 2023-10-21T04:15:37.642 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/4093318189 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\4\0\9\3\3\1\8\1\8\9 ]] 2023-10-21T04:15:37.642 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:15:47.645 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:47.646 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:15:47.647 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:47.668 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:48.544 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/4093318189 2023-10-21T04:15:48.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:15:48.545 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861748 2023-10-21T04:15:48.545 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:48.545 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/4093318189 ]] 2023-10-21T04:15:48.546 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/4093318189 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\4\0\9\3\3\1\8\1\8\9 ]] 2023-10-21T04:15:48.546 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:15:58.546 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:15:58.547 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:58.547 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:58.571 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:59.457 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/4093318189 2023-10-21T04:15:59.457 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:15:59.458 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861759 2023-10-21T04:15:59.458 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:59.458 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/4093318189 ]] 2023-10-21T04:15:59.458 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/4093318189 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\4\0\9\3\3\1\8\1\8\9 ]] 2023-10-21T04:15:59.459 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:16:09.463 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:09.466 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:09.469 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:16:09.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:10.520 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1214594455 2023-10-21T04:16:10.520 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:16:10.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861770 2023-10-21T04:16:10.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:10.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1214594455 ]] 2023-10-21T04:16:10.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1214594455 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\4\0\9\3\3\1\8\1\8\9 ]] 2023-10-21T04:16:10.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/1214594455 2023-10-21T04:16:12.209 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/1214594455 until 2023-10-21T05:16:11.391445+0000 (3600 sec) 2023-10-21T04:16:12.270 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:16:12.271 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/1214594455 2023-10-21T04:16:13.057 INFO:tasks.workunit.client.0.smithi119.stderr:listed 9 entries 2023-10-21T04:16:13.111 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/1214594455 2023-10-21T04:16:13.112 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:16:23.113 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:16:23.115 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:23.116 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:23.131 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:24.061 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1214594455 2023-10-21T04:16:24.062 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:16:24.063 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861784 2023-10-21T04:16:24.063 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:24.063 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1214594455 ]] 2023-10-21T04:16:24.063 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1214594455 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\1\4\5\9\4\4\5\5 ]] 2023-10-21T04:16:24.064 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:16:34.066 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:34.067 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:16:34.069 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:34.082 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:34.351 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:16:34.352+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:16:34.355 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:16:34.356+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:16:34.355 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:16:34.356+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:16:34.355 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:16:34.356+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T04:16:34.355 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:16:34.356+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:16:34.355 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:16:34.356+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:16:34.356 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:16:34.356+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:16:34.356 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:16:34.356+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:16:34.356 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:16:34.356+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:16:34.356 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:16:34.356+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:16:34.941 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1214594455 2023-10-21T04:16:34.942 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:16:34.942 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861794 2023-10-21T04:16:34.942 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:34.942 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1214594455 ]] 2023-10-21T04:16:34.943 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1214594455 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\1\4\5\9\4\4\5\5 ]] 2023-10-21T04:16:34.943 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:16:44.948 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:16:44.951 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:44.952 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:44.966 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:45.929 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1214594455 2023-10-21T04:16:45.930 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:16:45.930 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861805 2023-10-21T04:16:45.930 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:45.930 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1214594455 ]] 2023-10-21T04:16:45.931 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1214594455 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\1\4\5\9\4\4\5\5 ]] 2023-10-21T04:16:45.931 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:16:55.933 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:55.933 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:55.934 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:16:55.935 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:56.786 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1214594455 2023-10-21T04:16:56.787 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:16:56.788 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861816 2023-10-21T04:16:56.788 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:56.788 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1214594455 ]] 2023-10-21T04:16:56.788 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1214594455 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\1\4\5\9\4\4\5\5 ]] 2023-10-21T04:16:56.788 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:16:59.999 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:16:59.999+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:59.999 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:16:59.999+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e09b1100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.000 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.003+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.000 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.003+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e09b1300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.005 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.007+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.005 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.007+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e09b1500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.011 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.011+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.011 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.011+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e09b1700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.019 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.019+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.019 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.019+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e09b1900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.039 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.039+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.039 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.039+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e09b1b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.046 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.047+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.047 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.047+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e09b1d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.059 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.059+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.059 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.059+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e09b1f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.065 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.067+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.065 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.067+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0e46100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.071 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.071+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.072 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.071+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0e46300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.079 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.079+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.080 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.079+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0e46500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.082 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.083+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.082 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.083+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0e46700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.087 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.087+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.087 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.091+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0e46900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.094 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.095+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.095 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.095+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0e46b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.101 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.103+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.101 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.103+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0e46d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.107 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.107+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.107 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.107+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0e46f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.111 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.111+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.111 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.111+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0e47100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.120 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.123+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.120 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.123+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0e47300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.127 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.127+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.127 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.127+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0e47500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.133 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.135+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.133 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:17:00.135+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0e47700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:06.791 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:06.792 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:17:06.799 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:06.802 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:07.689 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1214594455 2023-10-21T04:17:07.714 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:17:07.715 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861827 2023-10-21T04:17:07.715 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:07.716 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1214594455 ]] 2023-10-21T04:17:07.716 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1214594455 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\1\4\5\9\4\4\5\5 ]] 2023-10-21T04:17:07.716 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:17:17.723 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:17.724 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:17:17.729 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:17.731 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:18.635 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1214594455 2023-10-21T04:17:18.636 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:17:18.637 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861838 2023-10-21T04:17:18.637 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:18.637 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1214594455 ]] 2023-10-21T04:17:18.637 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1214594455 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\1\4\5\9\4\4\5\5 ]] 2023-10-21T04:17:18.638 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:17:26.093 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:17:26.094+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T04:17:26.093 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:17:26.094+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:17:26.093 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:17:26.094+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:17:26.094 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:17:26.094+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:17:28.650 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:17:28.651 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:28.664 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:28.672 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:29.674 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1332372757 2023-10-21T04:17:29.675 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:17:29.675 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861849 2023-10-21T04:17:29.675 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:29.676 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1332372757 ]] 2023-10-21T04:17:29.676 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1332372757 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\2\1\4\5\9\4\4\5\5 ]] 2023-10-21T04:17:29.676 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/1332372757 2023-10-21T04:17:32.221 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/1332372757 until 2023-10-21T05:17:31.531871+0000 (3600 sec) 2023-10-21T04:17:32.251 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/1332372757 2023-10-21T04:17:32.252 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:17:32.994 INFO:tasks.workunit.client.0.smithi119.stderr:listed 10 entries 2023-10-21T04:17:33.022 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/1332372757 2023-10-21T04:17:33.022 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:17:43.029 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:43.030 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:17:43.033 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:43.044 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:43.996 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1332372757 2023-10-21T04:17:43.997 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:17:43.998 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861864 2023-10-21T04:17:43.998 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:43.998 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1332372757 ]] 2023-10-21T04:17:43.999 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1332372757 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\3\3\2\3\7\2\7\5\7 ]] 2023-10-21T04:17:43.999 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:17:54.001 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:54.002 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:17:54.004 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:54.023 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:54.880 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1332372757 2023-10-21T04:17:54.880 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:17:54.881 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861874 2023-10-21T04:17:54.881 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:54.881 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1332372757 ]] 2023-10-21T04:17:54.881 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1332372757 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\3\3\2\3\7\2\7\5\7 ]] 2023-10-21T04:17:54.881 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:17:56.066 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:17:56.069+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:17:56.069 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:17:56.069+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:17:56.070 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:17:56.069+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:17:56.070 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:17:56.069+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:18:00.002 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.005+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.002 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.005+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e47b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.004 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.009+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.005 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.009+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e47980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.009 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.013+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.010 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.013+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d80000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.013 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.017+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.013 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.017+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d80200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.022 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.025+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.023 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.025+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d80400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.026 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.029+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.026 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.029+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d80600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.029 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.033+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.030 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.033+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d80c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.033 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.037+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.034 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.037+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d80a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.037 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.041+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.037 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.041+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d81180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.041 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.045+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.041 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.045+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d81000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.052 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.053+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.052 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.053+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d80e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.058 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.061+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.058 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.061+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d80880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.080 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.085+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.080 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.085+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d81400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.084 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.089+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.085 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.089+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d81600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.098 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.101+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.098 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.101+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d81800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.101 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.105+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.101 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.105+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d81a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.106 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.109+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.106 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.109+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d81c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.114 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.113+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.114 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.113+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d81e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.116 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.117+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.116 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.117+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d80900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.129 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.133+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.129 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:18:00.133+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d81080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:04.882 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:18:04.883 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:04.883 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:04.884 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:05.840 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1332372757 2023-10-21T04:18:05.842 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:18:05.843 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861885 2023-10-21T04:18:05.843 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:05.843 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1332372757 ]] 2023-10-21T04:18:05.843 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1332372757 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\3\3\2\3\7\2\7\5\7 ]] 2023-10-21T04:18:05.843 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:18:15.846 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:18:15.846 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:15.846 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:15.851 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:16.728 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1332372757 2023-10-21T04:18:16.728 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:18:16.729 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861896 2023-10-21T04:18:16.729 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:16.729 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1332372757 ]] 2023-10-21T04:18:16.729 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1332372757 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\3\3\2\3\7\2\7\5\7 ]] 2023-10-21T04:18:16.730 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:18:26.737 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:18:26.743 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:26.745 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:26.745 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:27.673 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1332372757 2023-10-21T04:18:27.674 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:18:27.674 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861907 2023-10-21T04:18:27.674 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:27.675 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1332372757 ]] 2023-10-21T04:18:27.675 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1332372757 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\3\3\2\3\7\2\7\5\7 ]] 2023-10-21T04:18:27.675 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:18:37.682 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:18:37.682 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:37.683 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:37.687 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:38.607 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1485658198 2023-10-21T04:18:38.607 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:18:38.607 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861918 2023-10-21T04:18:38.608 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:38.608 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1485658198 ]] 2023-10-21T04:18:38.608 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1485658198 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\3\3\2\3\7\2\7\5\7 ]] 2023-10-21T04:18:38.608 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/1485658198 2023-10-21T04:18:40.251 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/1485658198 until 2023-10-21T05:18:39.507375+0000 (3600 sec) 2023-10-21T04:18:40.296 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:18:40.296 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/1485658198 2023-10-21T04:18:41.096 INFO:tasks.workunit.client.0.smithi119.stderr:listed 11 entries 2023-10-21T04:18:41.124 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/1485658198 2023-10-21T04:18:41.124 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:18:48.174 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:18:48.176+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:18:48.178 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:18:48.180+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:18:48.178 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:18:48.180+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:18:48.179 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:18:48.180+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:18:48.179 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:18:48.180+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:18:48.179 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:18:48.180+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:18:48.179 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:18:48.180+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:18:48.179 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:18:48.180+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:18:48.179 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:18:48.180+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:18:51.127 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:18:51.127 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:51.128 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:51.128 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:52.085 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1485658198 2023-10-21T04:18:52.085 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:18:52.086 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861932 2023-10-21T04:18:52.086 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:52.087 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1485658198 ]] 2023-10-21T04:18:52.087 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1485658198 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\4\8\5\6\5\8\1\9\8 ]] 2023-10-21T04:18:52.087 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:19:00.001 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.003+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.002 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.003+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e09d9e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.004 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.003+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.004 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.003+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d0cd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.013 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.015+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.013 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.015+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d0d500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.022 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.023+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.022 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.023+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d0dd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.031 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.031+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.031 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.031+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d0cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.039 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.039+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.039 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.039+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e47f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.047 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.047+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.047 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.047+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d80780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.053 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.055+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.053 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.055+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d80500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.058 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.059+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.058 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.059+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d80e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.062 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.063+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.062 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.063+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d81280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.064 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.063+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.064 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.063+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d81480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.067 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.067+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.068 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.067+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e46680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.071 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.071+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.071 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.071+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d81600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.073 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.075+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.074 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.075+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d81800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.078 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.079+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.078 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.079+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d81a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.083 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.083+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.083 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.083+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d81c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.086 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.087+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.087 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.087+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e36180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.091 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.091+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.091 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.091+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e36380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.095 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.095+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.095 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.095+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e36580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.100 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.099+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.100 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:19:00.099+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e36780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:02.087 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:19:02.093 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:02.095 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:02.108 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:02.981 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1485658198 2023-10-21T04:19:02.981 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:19:02.983 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861942 2023-10-21T04:19:02.983 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:02.983 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1485658198 ]] 2023-10-21T04:19:02.983 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1485658198 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\4\8\5\6\5\8\1\9\8 ]] 2023-10-21T04:19:02.984 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:19:12.987 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:12.987 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:12.989 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:19:13.017 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:13.965 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1485658198 2023-10-21T04:19:13.965 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:19:13.966 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861953 2023-10-21T04:19:13.966 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:13.966 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1485658198 ]] 2023-10-21T04:19:13.967 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1485658198 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\4\8\5\6\5\8\1\9\8 ]] 2023-10-21T04:19:13.967 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:19:16.138 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:19:16.139+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:19:23.971 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:23.971 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:23.974 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:23.984 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:19:24.848 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1485658198 2023-10-21T04:19:24.852 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:19:24.853 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861964 2023-10-21T04:19:24.854 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:24.854 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1485658198 ]] 2023-10-21T04:19:24.854 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1485658198 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\4\8\5\6\5\8\1\9\8 ]] 2023-10-21T04:19:24.854 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:19:34.869 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:19:34.873 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:34.874 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:34.877 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:35.829 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/1485658198 2023-10-21T04:19:35.829 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:19:35.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861975 2023-10-21T04:19:35.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:35.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/1485658198 ]] 2023-10-21T04:19:35.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/1485658198 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\4\8\5\6\5\8\1\9\8 ]] 2023-10-21T04:19:35.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:19:36.725 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:19:36.726+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:19:36.725 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:19:36.726+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:19:45.851 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:45.854 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:19:45.859 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:45.862 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:46.863 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2726235268 2023-10-21T04:19:46.864 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:19:46.865 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697861986 2023-10-21T04:19:46.865 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:46.865 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2726235268 ]] 2023-10-21T04:19:46.865 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2726235268 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\4\8\5\6\5\8\1\9\8 ]] 2023-10-21T04:19:46.865 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/2726235268 2023-10-21T04:19:48.629 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/2726235268 until 2023-10-21T05:19:47.921832+0000 (3600 sec) 2023-10-21T04:19:48.661 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:19:48.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/2726235268 2023-10-21T04:19:49.384 INFO:tasks.workunit.client.0.smithi119.stderr:listed 12 entries 2023-10-21T04:19:49.411 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/2726235268 2023-10-21T04:19:49.412 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:19:53.502 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:19:53.506+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:19:53.502 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:19:53.506+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:19:53.503 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:19:53.506+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:19:53.503 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:19:53.506+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T04:19:53.503 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:19:53.506+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:19:53.503 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:19:53.506+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:19:53.503 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:19:53.506+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:19:53.503 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:19:53.506+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:19:53.504 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:19:53.506+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:19:53.504 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:19:53.506+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:19:59.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:19:59.428 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:59.429 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:59.432 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:00.003 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.002+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.003 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.002+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d0c380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.003 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.006+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.004 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.006+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0c06000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.007 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.010+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.007 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.010+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e37780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.016 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.018+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.016 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.018+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e36080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.027 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.030+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.028 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.030+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e37c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.037 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.042+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.038 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.042+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e37e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.041 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.046+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.041 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.046+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0dfe080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.051 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.054+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.051 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.054+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0dfe280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.056 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.058+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.056 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.058+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0dfe480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.062 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.066+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.062 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.066+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0dfe680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.067 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.070+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.068 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.070+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0dfe880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.074 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.078+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.074 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.078+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0dfea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.080 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.082+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.080 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.082+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0dff200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.086 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.090+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.086 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.090+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0dff080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.092 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.094+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.092 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.094+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0dfee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.098 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.102+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.098 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.102+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0dfed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.104 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.106+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.104 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.106+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0dff480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.112 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.114+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.112 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.114+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0dff680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.118 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.122+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.118 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.122+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0dff880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.123 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.126+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.124 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:20:00.126+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0dffa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.440 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2726235268 2023-10-21T04:20:00.441 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:20:00.441 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862000 2023-10-21T04:20:00.441 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:00.442 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2726235268 ]] 2023-10-21T04:20:00.442 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2726235268 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\2\6\2\3\5\2\6\8 ]] 2023-10-21T04:20:00.442 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:20:10.445 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:10.446 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:20:10.453 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:10.460 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:11.396 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2726235268 2023-10-21T04:20:11.397 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:20:11.398 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862011 2023-10-21T04:20:11.399 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:11.399 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2726235268 ]] 2023-10-21T04:20:11.399 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2726235268 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\2\6\2\3\5\2\6\8 ]] 2023-10-21T04:20:11.399 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:20:21.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:20:21.401 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:21.406 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:21.407 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:22.367 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2726235268 2023-10-21T04:20:22.368 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:20:22.368 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862022 2023-10-21T04:20:22.369 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:22.369 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2726235268 ]] 2023-10-21T04:20:22.369 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2726235268 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\2\6\2\3\5\2\6\8 ]] 2023-10-21T04:20:22.369 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:20:30.977 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:20:30.977+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T04:20:30.978 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:20:30.977+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130c30 2023-10-21T04:20:30.978 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:20:30.977+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:20:30.978 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:20:30.977+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:20:30.978 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:20:30.977+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:20:32.370 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:20:32.370 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:32.371 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:32.380 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:33.220 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2726235268 2023-10-21T04:20:33.220 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:20:33.221 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862033 2023-10-21T04:20:33.221 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:33.222 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2726235268 ]] 2023-10-21T04:20:33.222 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2726235268 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\2\6\2\3\5\2\6\8 ]] 2023-10-21T04:20:33.222 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:20:42.502 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:20:42.500+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:20:42.503 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:20:42.500+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:20:42.514 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:20:42.516+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:20:42.515 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:20:42.516+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:20:42.515 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:20:42.516+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:20:42.515 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:20:42.516+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:20:42.515 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:20:42.516+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:20:42.515 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:20:42.516+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:20:43.231 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:20:43.233 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:43.237 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:43.244 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:44.129 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/2726235268 2023-10-21T04:20:44.130 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:20:44.130 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862044 2023-10-21T04:20:44.131 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:44.131 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/2726235268 ]] 2023-10-21T04:20:44.131 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/2726235268 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\2\6\2\3\5\2\6\8 ]] 2023-10-21T04:20:44.131 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:20:49.890 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:20:49.892+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:20:49.891 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:20:49.892+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:20:49.891 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:20:49.892+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:20:54.132 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:54.134 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:20:54.135 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:54.148 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:55.125 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/843344947 2023-10-21T04:20:55.126 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:20:55.126 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862055 2023-10-21T04:20:55.126 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:55.127 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/843344947 ]] 2023-10-21T04:20:55.127 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/843344947 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\2\7\2\6\2\3\5\2\6\8 ]] 2023-10-21T04:20:55.127 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/843344947 2023-10-21T04:20:55.226 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:20:55.228+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:20:57.291 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/843344947 until 2023-10-21T05:20:56.585861+0000 (3600 sec) 2023-10-21T04:20:57.319 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:20:57.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/843344947 2023-10-21T04:20:57.989 INFO:tasks.workunit.client.0.smithi119.stderr:listed 13 entries 2023-10-21T04:20:58.018 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/843344947 2023-10-21T04:20:58.018 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:20:59.999 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.000+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:59.999 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.000+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0421480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.001 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.004+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.002 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.004+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db497880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.006 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.008+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.006 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.008+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e37800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.011 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.012+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.011 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.012+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625db496c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.020 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.020+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.021 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.020+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e37200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.026 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.028+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.026 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.028+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d81800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.033 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.032+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.033 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.032+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e46100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.037 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.036+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.037 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.040+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e37480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.060 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.060+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.060 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.060+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0d81c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.066 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.068+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.066 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.068+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0dffc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.081 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.084+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.081 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.084+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0dffe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.093 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.096+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.093 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.096+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0e46980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.098 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.100+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.098 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.100+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0a17100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.106 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.108+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.107 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.108+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0a17300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.112 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.112+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.112 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.112+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0a17500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.117 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.116+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.118 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.120+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0a17700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.122 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.124+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.122 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.124+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0a17900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.126 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.128+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.126 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.128+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0a17b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.130 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.132+0000 7ffacc1d7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.130 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.132+0000 7ffacc1d7640 -1 librbd::ImageState: 0x5625e0a17180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.135 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.136+0000 7ffacb9d6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.135 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:21:00.136+0000 7ffacb9d6640 -1 librbd::ImageState: 0x5625e0a17d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:08.025 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:08.026 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:08.026 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:08.027 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:21:08.933 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/843344947 2023-10-21T04:21:08.933 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:21:08.934 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862068 2023-10-21T04:21:08.934 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:08.934 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/843344947 ]] 2023-10-21T04:21:08.934 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/843344947 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\8\4\3\3\4\4\9\4\7 ]] 2023-10-21T04:21:08.935 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:21:18.935 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:21:18.937 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:18.937 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:18.941 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:19.835 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/843344947 2023-10-21T04:21:19.836 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:21:19.836 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862079 2023-10-21T04:21:19.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:19.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/843344947 ]] 2023-10-21T04:21:19.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/843344947 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\8\4\3\3\4\4\9\4\7 ]] 2023-10-21T04:21:19.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:21:25.114 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:21:25.115+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:21:29.841 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:21:29.842 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:29.842 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:29.846 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:30.786 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/843344947 2023-10-21T04:21:30.787 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:21:30.788 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862090 2023-10-21T04:21:30.788 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:30.788 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/843344947 ]] 2023-10-21T04:21:30.788 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/843344947 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\8\4\3\3\4\4\9\4\7 ]] 2023-10-21T04:21:30.788 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:21:32.226 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:21:32.227+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:21:40.789 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:21:40.790 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:40.791 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:40.795 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:41.690 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/843344947 2023-10-21T04:21:41.691 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:21:41.691 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862101 2023-10-21T04:21:41.691 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:41.691 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/843344947 ]] 2023-10-21T04:21:41.692 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/843344947 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\8\4\3\3\4\4\9\4\7 ]] 2023-10-21T04:21:41.692 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:21:42.870 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:21:42.863+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:21:42.871 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:21:42.863+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:21:51.694 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:21:51.694 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:51.697 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:51.710 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:52.605 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/843344947 2023-10-21T04:21:52.606 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:21:52.606 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862112 2023-10-21T04:21:52.606 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:52.607 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/843344947 ]] 2023-10-21T04:21:52.607 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/843344947 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\8\4\3\3\4\4\9\4\7 ]] 2023-10-21T04:21:52.607 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:22:02.611 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:02.611 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:02.612 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:22:02.612 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:03.556 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/843344947 2023-10-21T04:22:03.557 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:22:03.560 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862123 2023-10-21T04:22:03.560 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:03.560 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/843344947 ]] 2023-10-21T04:22:03.561 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/843344947 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\8\4\3\3\4\4\9\4\7 ]] 2023-10-21T04:22:03.561 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:22:13.585 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:22:13.586 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:13.586 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:13.587 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:14.799 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:22:14.800 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:22:14.810 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862134 2023-10-21T04:22:14.810 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:14.810 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:22:14.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\8\4\3\3\4\4\9\4\7 ]] 2023-10-21T04:22:14.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist add 172.21.15.119:0/161644020 2023-10-21T04:22:17.319 INFO:tasks.workunit.client.0.smithi119.stderr:blocklisting 172.21.15.119:0/161644020 until 2023-10-21T05:22:16.541422+0000 (3600 sec) 2023-10-21T04:22:17.677 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd blocklist ls 2023-10-21T04:22:17.679 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep -q 172.21.15.119:0/161644020 2023-10-21T04:22:18.438 INFO:tasks.workunit.client.0.smithi119.stderr:listed 14 entries 2023-10-21T04:22:18.462 INFO:tasks.workunit.client.0.smithi119.stderr:+ PREV_CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:22:18.463 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:22:19.186 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:22:19.185+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:22:19.186 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:22:19.185+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:22:28.585 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:22:28.585 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:28.586 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:28.605 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:29.627 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:22:29.632 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:22:29.633 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862149 2023-10-21T04:22:29.633 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:29.633 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:22:29.634 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:22:29.634 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:22:39.659 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:39.660 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:22:39.660 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:39.660 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:40.746 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:22:40.746 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:22:40.748 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862160 2023-10-21T04:22:40.748 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:40.748 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:22:40.749 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:22:40.749 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:22:50.754 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:50.755 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:22:50.758 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:50.760 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:51.796 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:22:51.796 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:22:51.797 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862171 2023-10-21T04:22:51.797 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:51.797 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:22:51.798 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:22:51.798 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:23:00.001 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.000+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.001 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.000+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e06c8f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.004 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.004+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.004 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.004+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e09b1e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.011 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.012+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.011 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.012+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625dfcdc300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.026 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.028+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.026 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.028+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e09b1600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.033 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.032+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.033 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.032+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0e47580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.044 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.044+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.044 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.044+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0e47600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.048 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.048+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.048 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.048+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0e46600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.053 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.052+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.053 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.052+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0e47d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.068 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.068+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.068 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.068+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0e37180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.083 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.084+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.083 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.084+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0e36e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.091 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.088+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.091 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.088+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0e47500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.103 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.104+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.103 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.104+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0e46a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.109 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.108+0000 7ffacb1d5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:00.110 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T04:23:00.108+0000 7ffacb1d5640 -1 librbd::ImageState: 0x5625e0d81a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:23:01.604 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:23:01.604+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:23:01.808 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:23:01.809 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:01.814 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:01.818 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:02.896 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:23:02.905 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:23:02.905 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862182 2023-10-21T04:23:02.906 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:02.906 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:23:02.906 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:23:02.906 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:23:05.851 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:23:05.848+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:23:05.851 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:23:05.848+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:23:05.851 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:23:05.848+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:23:12.907 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:23:12.912 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:12.914 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:12.919 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:13.845 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:23:13.845 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:23:13.846 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862193 2023-10-21T04:23:13.846 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:13.846 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:23:13.846 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:23:13.847 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:23:23.856 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:23.860 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:23:23.870 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:23.876 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:24.818 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:23:24.818 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:23:24.819 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862204 2023-10-21T04:23:24.819 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:24.819 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:23:24.819 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:23:24.820 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:23:34.829 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:34.830 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:34.830 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:34.857 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:23:35.862 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:23:35.863 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:23:35.863 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862215 2023-10-21T04:23:35.863 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:35.864 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:23:35.864 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:23:35.864 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:23:36.708 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:23:36.707+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T04:23:36.709 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:23:36.707+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:23:36.709 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:23:36.707+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:23:36.709 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:23:36.707+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:23:45.865 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:45.866 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:23:45.873 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:45.874 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:46.858 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:23:46.858 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:23:46.859 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862226 2023-10-21T04:23:46.859 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:46.860 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:23:46.860 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:23:46.860 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:23:48.313 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:23:48.307+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:23:48.313 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:23:48.307+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:23:52.625 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:23:52.626+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:23:52.626 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:23:52.626+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:23:52.626 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:23:52.626+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:23:52.626 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:23:52.626+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:23:52.626 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:23:52.626+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:23:52.627 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:23:52.626+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:23:56.866 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:23:56.867 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:56.872 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:56.873 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:57.724 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:23:57.724 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:23:57.725 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862237 2023-10-21T04:23:57.725 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:57.725 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:23:57.725 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:23:57.725 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:24:07.730 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:24:07.731 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:07.731 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:07.737 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:08.718 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:24:08.719 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:24:08.719 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862248 2023-10-21T04:24:08.719 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:08.720 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:24:08.720 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:24:08.720 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:24:18.725 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:24:18.725 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:18.727 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:18.739 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:19.722 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:24:19.727 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:24:19.728 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862259 2023-10-21T04:24:19.728 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:19.728 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:24:19.728 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:24:19.729 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:24:29.737 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:29.737 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:29.737 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:24:29.746 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:30.745 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:24:30.745 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:24:30.746 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862270 2023-10-21T04:24:30.746 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:30.746 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:24:30.746 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:24:30.747 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:24:40.747 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:24:40.751 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:40.751 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:40.751 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:41.433 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:24:41.425+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:24:41.437 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:24:41.437+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:24:41.437 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:24:41.437+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:24:41.437 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:24:41.437+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:24:41.437 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:24:41.437+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:24:41.437 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:24:41.437+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:24:41.438 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:24:41.437+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:24:41.438 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:24:41.437+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:24:41.438 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:24:41.437+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:24:41.666 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:24:41.667 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:24:41.667 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862281 2023-10-21T04:24:41.668 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:41.668 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:24:41.668 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:24:41.668 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:24:51.672 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:51.672 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:51.672 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:24:51.675 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:52.603 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:24:52.603 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:24:52.604 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862292 2023-10-21T04:24:52.604 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:52.604 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:24:52.604 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:24:52.604 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:25:02.606 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:25:02.612 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:02.613 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:02.615 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:03.442 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:25:03.442 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:25:03.443 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862303 2023-10-21T04:25:03.443 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:03.443 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:25:03.444 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:25:03.444 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:25:13.446 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:25:13.447 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:13.447 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:13.449 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:14.486 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:25:14.486 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:25:14.487 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862314 2023-10-21T04:25:14.487 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:14.487 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:25:14.487 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:25:14.487 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:25:22.574 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:25:22.572+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:25:22.575 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:25:22.572+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:25:24.491 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:25:24.497 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:24.498 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:24.500 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:25.365 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:25:25.366 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:25:25.366 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862325 2023-10-21T04:25:25.367 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:25.367 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:25:25.367 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:25:25.367 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:25:35.377 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:35.377 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:35.378 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:25:35.381 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:36.440 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:25:36.441 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:25:36.442 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862336 2023-10-21T04:25:36.442 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:36.443 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:25:36.443 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:25:36.443 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:25:46.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:25:46.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:46.448 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:46.460 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:47.386 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:25:47.387 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:25:47.387 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862347 2023-10-21T04:25:47.387 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:47.388 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:25:47.388 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:25:47.388 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:25:48.099 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:25:48.099+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:25:57.390 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:25:57.391 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:57.391 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:57.399 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:58.237 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:25:58.243 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:25:58.245 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862358 2023-10-21T04:25:58.245 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:58.245 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:25:58.245 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:25:58.245 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:26:08.247 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:26:08.247 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:08.248 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:08.252 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:09.178 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:26:09.178 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:26:09.179 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862369 2023-10-21T04:26:09.179 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:09.179 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:26:09.180 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:26:09.180 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:26:19.186 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:26:19.186 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:19.187 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:19.187 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:20.121 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:26:20.121 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:26:20.122 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862380 2023-10-21T04:26:20.122 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:20.122 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:26:20.122 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:26:20.123 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:26:20.291 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:26:20.291+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:26:20.291 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:26:20.291+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:26:30.125 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:30.126 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:26:30.136 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:30.146 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:31.143 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:26:31.146 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:26:31.147 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862391 2023-10-21T04:26:31.147 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:31.148 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:26:31.148 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:26:31.148 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:26:32.919 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:26:32.914+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:26:32.919 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:26:32.914+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:26:32.920 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:26:32.914+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:26:32.920 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:26:32.914+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:26:32.920 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:26:32.914+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:26:32.921 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:26:32.914+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:26:41.150 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:26:41.150 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:41.151 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:41.177 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:42.120 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:26:42.121 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:26:42.121 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862402 2023-10-21T04:26:42.122 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:42.122 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:26:42.122 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:26:42.122 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:26:52.124 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:52.127 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:52.129 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:52.135 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:26:53.065 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:26:53.065 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:26:53.066 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862413 2023-10-21T04:26:53.066 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:53.066 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:26:53.066 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:26:53.067 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:26:58.005 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:26:58.006+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:27:03.067 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:27:03.070 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:03.077 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:03.079 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:04.038 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:27:04.044 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:27:04.052 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862424 2023-10-21T04:27:04.052 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:04.053 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:27:04.053 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:27:04.053 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:27:14.054 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:27:14.054 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:14.055 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:14.057 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:14.958 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:27:14.958 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:27:14.959 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862434 2023-10-21T04:27:14.959 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:14.959 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:27:14.959 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:27:14.959 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:27:24.962 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:24.962 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:24.963 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:24.963 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:27:25.890 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:27:25.890 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:27:25.891 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862445 2023-10-21T04:27:25.891 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:25.891 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:27:25.892 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:27:25.892 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:27:27.722 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:27.717+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:27:27.730 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:27.729+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:27:27.730 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:27.729+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:27:27.731 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:27.729+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T04:27:27.731 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:27.729+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:27:27.731 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:27.729+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:27:27.731 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:27.729+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:27:27.732 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:27.729+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:27:27.732 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:27.729+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:27:27.732 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:27.729+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:27:35.893 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:27:35.895 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:35.895 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:35.895 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:36.785 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:27:36.786 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:27:36.787 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862456 2023-10-21T04:27:36.787 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:36.787 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:27:36.787 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:27:36.788 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:27:46.804 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:27:46.805 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:46.808 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:46.809 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:47.418 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:47.417+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:27:47.434 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:47.433+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:27:47.434 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:47.433+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:27:47.434 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:47.433+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T04:27:47.434 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:47.433+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:27:47.435 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:47.433+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:27:47.435 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:47.433+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:27:47.435 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:47.433+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:27:47.435 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:47.433+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:27:47.436 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:47.433+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:27:47.751 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:27:47.752 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:27:47.752 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862467 2023-10-21T04:27:47.752 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:47.753 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:27:47.753 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:27:47.753 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:27:54.731 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:54.729+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:27:54.731 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:27:54.729+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:27:57.754 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:57.755 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:57.758 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:27:57.769 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:58.868 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:27:58.869 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:27:58.873 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862478 2023-10-21T04:27:58.873 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:58.873 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:27:58.873 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:27:58.874 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:28:02.258 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:28:02.253+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:28:02.259 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:28:02.253+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:28:02.259 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:28:02.253+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:28:02.259 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:28:02.253+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:28:02.260 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:28:02.253+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:28:02.260 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:28:02.253+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:28:02.260 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:28:02.261+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:28:08.636 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:28:08.632+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:28:08.636 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:28:08.636+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:28:08.637 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:28:08.636+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:28:08.637 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:28:08.636+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:28:08.879 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:08.880 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:08.880 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:28:08.884 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:09.833 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:28:09.834 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:28:09.835 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862489 2023-10-21T04:28:09.835 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:09.835 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:28:09.835 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:28:09.836 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:28:19.847 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:19.849 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:28:19.858 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:19.869 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:20.836 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:28:20.836 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:28:20.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862500 2023-10-21T04:28:20.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:20.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:28:20.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:28:20.838 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:28:30.840 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:30.840 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:30.850 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:30.854 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:28:31.682 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:28:31.682 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:28:31.683 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862511 2023-10-21T04:28:31.683 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:31.683 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:28:31.683 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:28:31.684 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:28:41.688 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:28:41.688 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:41.689 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:41.689 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:42.610 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:28:42.614 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:28:42.615 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862522 2023-10-21T04:28:42.616 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:42.616 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:28:42.616 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:28:42.616 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:28:52.617 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:28:52.618 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:52.622 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:52.628 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:53.424 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:28:53.425 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:28:53.425 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862533 2023-10-21T04:28:53.425 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:53.426 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:28:53.426 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:28:53.426 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:29:03.428 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:03.429 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:29:03.429 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:03.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:04.392 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:29:04.392 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:29:04.392 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862544 2023-10-21T04:29:04.393 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:04.393 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:29:04.393 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:29:04.393 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:29:14.421 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:14.434 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:14.437 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:29:14.445 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:15.288 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:29:15.290 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:29:15.290 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862555 2023-10-21T04:29:15.291 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:15.291 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:29:15.291 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:29:15.292 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:29:25.294 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:29:25.295 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:25.295 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:25.296 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:26.278 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:29:26.278 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:29:26.279 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862566 2023-10-21T04:29:26.279 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:26.280 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:29:26.280 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:29:26.280 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:29:36.281 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:36.282 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:36.285 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:29:36.286 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:37.154 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:29:37.155 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:29:37.155 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862577 2023-10-21T04:29:37.155 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:37.156 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:29:37.156 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:29:37.156 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:29:47.163 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:29:47.170 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:47.171 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:47.200 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:48.054 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:29:48.058 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:29:48.059 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862588 2023-10-21T04:29:48.059 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:48.060 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:29:48.060 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:29:48.060 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:29:48.084 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:29:48.083+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:29:48.085 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:29:48.083+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:29:48.085 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:29:48.083+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:29:48.085 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:29:48.083+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:29:48.085 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:29:48.083+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:29:48.085 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:29:48.083+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:29:48.087 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:29:48.087+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:29:58.073 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:58.074 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:29:58.082 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:58.086 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:59.034 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:29:59.034 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:29:59.035 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862599 2023-10-21T04:29:59.035 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:59.036 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:29:59.036 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:29:59.036 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:30:09.037 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:30:09.037 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:09.037 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:09.038 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:09.968 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:30:09.968 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:30:09.968 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862609 2023-10-21T04:30:09.969 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:09.969 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:30:09.969 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:30:09.969 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:30:19.982 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:30:19.984 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:19.989 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:19.996 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:20.998 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:30:20.998 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:30:20.999 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862621 2023-10-21T04:30:20.999 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:20.999 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:30:20.999 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:30:21.000 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:30:31.000 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:30:31.001 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:31.006 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:31.012 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:31.865 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:30:31.866 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:30:31.867 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862631 2023-10-21T04:30:31.867 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:31.867 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:30:31.867 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:30:31.867 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:30:41.868 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:30:41.869 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:41.869 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:41.874 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:42.711 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:30:42.712 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:30:42.712 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862642 2023-10-21T04:30:42.712 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:42.712 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:30:42.713 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:30:42.713 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:30:52.717 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:30:52.717 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:52.725 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:52.734 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:53.829 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:30:53.829 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:30:53.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862653 2023-10-21T04:30:53.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:53.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:30:53.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:30:53.831 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:31:03.838 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:03.839 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:03.842 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:31:03.862 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:04.777 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:31:04.778 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:31:04.778 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862664 2023-10-21T04:31:04.779 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:04.779 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:31:04.779 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:31:04.779 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:31:14.781 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:31:14.782 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:14.783 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:14.799 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:15.828 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:31:15.829 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:31:15.829 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862675 2023-10-21T04:31:15.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:15.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:31:15.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:31:15.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:31:17.569 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:31:17.564+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:31:17.570 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:31:17.564+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:31:25.831 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:25.838 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:31:25.840 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:25.844 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:26.739 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:31:26.741 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:31:26.742 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862686 2023-10-21T04:31:26.743 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:26.743 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:31:26.743 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:31:26.744 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:31:28.390 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:31:28.388+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:31:28.390 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:31:28.388+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:31:28.391 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:31:28.388+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:31:36.753 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:36.754 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:31:36.760 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:36.763 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:37.935 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:31:37.935 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:31:37.936 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862697 2023-10-21T04:31:37.936 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:37.936 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:31:37.936 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:31:37.936 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:31:47.938 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:47.944 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:31:47.951 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:47.954 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:48.962 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:31:48.962 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:31:48.963 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862708 2023-10-21T04:31:48.963 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:48.963 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:31:48.963 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:31:48.964 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:31:58.964 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:31:58.965 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:58.967 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:58.975 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:59.970 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:31:59.970 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:31:59.970 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862719 2023-10-21T04:31:59.971 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:59.971 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:31:59.971 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:31:59.971 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:32:08.449 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:32:08.447+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:32:08.453 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:32:08.451+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:32:08.453 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:32:08.451+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:32:09.971 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:09.977 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:09.984 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:09.985 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:32:10.799 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:32:10.799 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:32:10.800 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862730 2023-10-21T04:32:10.800 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:10.800 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:32:10.801 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:32:10.801 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:32:19.603 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:32:19.602+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:32:20.802 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:32:20.807 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:20.807 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:20.822 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:21.744 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:32:21.749 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:32:21.750 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862741 2023-10-21T04:32:21.751 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:21.751 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:32:21.751 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:32:21.751 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:32:31.755 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:32:31.756 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:31.763 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:31.775 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:32.657 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:32:32.657 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:32:32.658 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862752 2023-10-21T04:32:32.658 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:32.658 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:32:32.659 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:32:32.659 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:32:42.660 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:42.661 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:42.664 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:42.665 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:32:43.632 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:32:43.632 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:32:43.635 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862763 2023-10-21T04:32:43.636 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:43.636 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:32:43.636 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:32:43.636 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:32:53.640 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:53.640 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:53.640 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:53.641 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:32:54.507 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:32:54.507 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:32:54.508 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862774 2023-10-21T04:32:54.508 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:54.508 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:32:54.508 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:32:54.508 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:32:57.316 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:32:57.317+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:32:57.316 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:32:57.317+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:32:57.316 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:32:57.317+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:33:04.509 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:33:04.510 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:04.515 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:04.535 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:05.389 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:33:05.389 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:33:05.390 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862785 2023-10-21T04:33:05.390 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:05.391 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:33:05.391 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:33:05.391 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:33:14.221 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:14.220+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:33:14.239 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:14.236+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:33:14.240 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:14.236+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:33:14.240 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:14.236+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T04:33:14.240 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:14.236+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:33:14.240 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:14.236+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:33:14.240 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:14.236+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:33:14.241 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:14.236+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:33:14.241 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:14.236+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:33:14.241 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:14.236+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:33:15.391 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:15.392 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:15.397 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:15.398 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:33:16.225 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:33:16.225 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:33:16.226 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862796 2023-10-21T04:33:16.226 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:16.226 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:33:16.226 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:33:16.230 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:33:24.304 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:33:24.300+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:33:24.305 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:33:24.300+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:33:24.305 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:33:24.300+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:33:26.229 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:26.229 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:26.231 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:26.235 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:33:27.143 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:33:27.144 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:33:27.145 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862807 2023-10-21T04:33:27.145 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:27.145 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:33:27.146 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:33:27.146 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:33:27.404 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:27.400+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:33:27.405 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:27.404+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:33:27.405 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:33:27.404+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:33:37.147 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:33:37.147 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:37.148 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:37.150 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:38.011 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:33:38.011 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:33:38.012 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862818 2023-10-21T04:33:38.012 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:38.012 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:33:38.012 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:33:38.013 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:33:38.286 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:33:38.280+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:33:48.013 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:33:48.015 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:48.020 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:48.023 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:48.868 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:33:48.868 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:33:48.869 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862828 2023-10-21T04:33:48.869 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:48.869 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:33:48.869 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:33:48.870 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:33:58.872 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:58.872 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:58.884 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:33:58.896 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:59.738 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:33:59.739 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:33:59.739 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862839 2023-10-21T04:33:59.739 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:59.740 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:33:59.740 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:33:59.740 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:34:01.263 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:34:01.259+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:34:01.263 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:34:01.259+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:34:09.745 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:09.746 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:09.746 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:34:09.755 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:10.739 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:34:10.739 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:34:10.740 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862850 2023-10-21T04:34:10.740 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:10.740 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:34:10.740 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:34:10.741 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:34:20.742 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:20.742 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:34:20.763 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:20.774 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:21.686 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:34:21.686 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:34:21.687 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862861 2023-10-21T04:34:21.688 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:21.688 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:34:21.688 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:34:21.688 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:34:31.694 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:34:31.699 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:31.699 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:31.703 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:32.566 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:34:32.567 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:34:32.567 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862872 2023-10-21T04:34:32.567 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:32.568 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:34:32.568 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:34:32.568 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:34:42.572 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:34:42.575 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:42.598 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:42.599 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:43.420 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862883 2023-10-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:34:46.339 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:46.338+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:34:48.347 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:48.346+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:34:48.348 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:48.346+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:34:48.348 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:48.346+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:34:48.348 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:48.346+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:34:49.674 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:49.670+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:34:49.675 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:49.670+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:34:49.683 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:49.682+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:34:53.432 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:34:53.432 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:53.433 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:53.433 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:54.401 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:34:54.401 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:34:54.402 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862894 2023-10-21T04:34:54.402 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:54.403 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:34:54.403 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:34:54.403 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:34:59.634 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:59.633+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:34:59.640 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:59.641+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:34:59.640 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:59.641+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:34:59.641 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:59.641+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:34:59.641 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:59.641+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:34:59.641 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:59.641+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:34:59.641 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:59.641+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:34:59.641 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:59.641+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:34:59.644 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:34:59.645+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:35:04.404 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:35:04.404 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:04.406 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:04.411 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:05.413 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:35:05.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:35:05.414 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862905 2023-10-21T04:35:05.414 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:05.414 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:35:05.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:35:05.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:35:15.418 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:15.418 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:15.419 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:35:15.437 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:16.292 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:35:16.292 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:35:16.293 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862916 2023-10-21T04:35:16.293 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:16.294 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:35:16.294 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:35:16.294 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:35:26.299 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:26.303 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:26.316 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:26.326 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:35:27.170 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:35:27.170 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:35:27.171 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862927 2023-10-21T04:35:27.171 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:27.171 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:35:27.171 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:35:27.171 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:35:32.738 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:35:32.737+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:35:32.739 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:35:32.737+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:35:37.186 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:35:37.198 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:37.198 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:37.200 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:38.084 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:35:38.085 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:35:38.085 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862938 2023-10-21T04:35:38.086 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:38.086 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:35:38.086 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:35:38.086 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:35:48.088 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:35:48.089 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:48.090 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:48.093 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:48.953 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:35:48.953 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:35:48.954 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862948 2023-10-21T04:35:48.954 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:48.955 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:35:48.955 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:35:48.955 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:35:58.959 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:35:58.960 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:58.969 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:58.970 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:59.746 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:35:59.747 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:35:59.747 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862959 2023-10-21T04:35:59.747 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:59.748 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:35:59.748 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:35:59.748 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:36:09.749 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:09.764 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:36:09.764 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:09.767 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:10.683 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:36:10.683 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:36:10.684 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862970 2023-10-21T04:36:10.684 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:10.685 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:36:10.685 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:36:10.685 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:36:11.614 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:36:11.608+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:36:11.615 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:36:11.612+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:36:11.628 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:36:11.616+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:36:11.628 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:36:11.616+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:36:11.628 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:36:11.616+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:36:11.629 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:36:11.616+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:36:20.696 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:36:20.697 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:20.699 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:20.708 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:21.601 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:36:21.602 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:36:21.602 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862981 2023-10-21T04:36:21.602 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:21.603 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:36:21.603 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:36:21.603 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:36:31.605 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:31.606 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:31.606 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:31.617 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:36:32.529 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:36:32.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:36:32.531 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697862992 2023-10-21T04:36:32.531 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:32.531 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:36:32.531 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:36:32.531 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:36:42.535 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:42.536 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:42.536 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:36:42.543 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:43.394 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:36:43.394 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:36:43.395 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863003 2023-10-21T04:36:43.395 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:43.395 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:36:43.395 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:36:43.396 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:36:53.262 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:36:53.259+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:36:53.405 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:53.405 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:53.408 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:53.408 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:36:54.346 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:36:54.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:36:54.346 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863014 2023-10-21T04:36:54.347 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:54.347 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:36:54.347 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:36:54.347 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:36:59.965 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:36:59.963+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:36:59.966 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:36:59.963+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:37:04.359 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:04.360 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:04.365 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:04.370 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:37:05.291 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:37:05.291 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:37:05.292 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863025 2023-10-21T04:37:05.292 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:05.292 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:37:05.292 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:37:05.293 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:37:15.313 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:15.314 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:37:15.324 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:15.329 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:16.352 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:37:16.352 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:37:16.353 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863036 2023-10-21T04:37:16.353 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:16.353 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:37:16.353 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:37:16.354 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:37:21.554 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:37:21.550+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:37:21.554 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:37:21.550+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:37:21.554 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:37:21.550+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:37:26.364 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:37:26.365 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:26.365 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:26.366 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:27.303 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:37:27.304 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:37:27.304 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863047 2023-10-21T04:37:27.304 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:27.305 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:37:27.305 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:37:27.305 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:37:37.308 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:37.311 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:37.314 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:37:37.322 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:38.248 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:37:38.248 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:37:38.249 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863058 2023-10-21T04:37:38.249 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:38.250 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:37:38.250 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:37:38.250 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:37:40.508 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:37:40.510+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T04:37:40.509 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:37:40.510+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130c30 2023-10-21T04:37:40.509 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:37:40.510+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:37:40.509 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:37:40.510+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:37:40.509 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:37:40.510+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:37:47.322 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:37:47.317+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:37:47.322 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:37:47.317+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:37:48.251 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:48.251 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:48.251 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:48.252 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:37:49.119 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:37:49.119 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:37:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863069 2023-10-21T04:37:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:37:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:37:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:37:59.121 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:37:59.123 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:59.127 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:59.128 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:00.112 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:38:00.112 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:38:00.113 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863080 2023-10-21T04:38:00.113 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:00.114 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:38:00.114 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:38:00.114 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:38:10.115 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:10.116 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:38:10.120 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:10.127 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:10.998 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:38:11.006 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:38:11.008 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863091 2023-10-21T04:38:11.008 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:11.008 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:38:11.008 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:38:11.009 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:38:21.024 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:38:21.032 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:21.042 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:21.046 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:22.047 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:38:22.048 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:38:22.049 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863102 2023-10-21T04:38:22.049 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:22.049 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:38:22.049 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:38:22.050 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:38:32.056 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:32.056 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:32.056 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:32.057 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:38:32.517 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:38:32.517+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T04:38:32.518 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:38:32.517+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:38:32.518 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:38:32.517+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:38:32.522 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:38:32.521+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:38:33.051 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:38:33.053 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:38:33.054 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863113 2023-10-21T04:38:33.054 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:33.054 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:38:33.055 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:38:33.055 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:38:43.055 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:38:43.056 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:43.064 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:43.065 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:43.924 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:38:43.925 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:38:43.925 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863123 2023-10-21T04:38:43.926 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:43.926 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:38:43.926 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:38:43.927 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:38:53.927 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:38:53.927 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:53.928 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:53.929 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:54.820 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:38:54.820 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:38:54.822 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863134 2023-10-21T04:38:54.822 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:54.822 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:38:54.822 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:38:54.822 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:39:01.801 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:39:01.800+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:39:04.826 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:04.826 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:39:04.826 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:04.842 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:05.733 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:39:05.733 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:39:05.734 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863145 2023-10-21T04:39:05.734 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:05.734 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:39:05.734 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:39:05.734 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:39:15.736 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:15.736 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:39:15.740 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:15.753 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:16.630 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:39:16.630 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:39:16.632 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863156 2023-10-21T04:39:16.632 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:16.632 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:39:16.632 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:39:16.632 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:39:26.636 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:26.638 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:39:26.641 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:26.643 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:27.583 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:39:27.584 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:39:27.585 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863167 2023-10-21T04:39:27.585 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:27.585 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:39:27.585 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:39:27.585 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:39:37.586 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:39:37.587 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:37.595 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:37.603 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:38.497 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:39:38.506 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:39:38.507 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863178 2023-10-21T04:39:38.507 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:38.507 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:39:38.507 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:39:38.507 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:39:42.593 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:42.587+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:39:42.606 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:42.591+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:39:42.606 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:42.591+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:39:42.607 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:42.591+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:39:42.607 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:42.591+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:39:42.607 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:42.591+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:39:42.607 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:42.595+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:39:48.508 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:39:48.509 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:48.518 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:48.518 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:49.500 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:39:49.501 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:39:49.501 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863189 2023-10-21T04:39:49.501 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:49.502 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:39:49.502 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:39:49.502 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:39:54.227 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:54.223+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:39:54.228 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:54.223+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:39:54.248 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:54.243+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:39:54.249 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:54.243+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:39:54.249 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:54.243+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:39:54.249 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:54.243+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:39:54.249 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:39:54.243+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:39:57.359 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:39:57.355+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:39:59.503 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:59.504 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:59.510 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:39:59.510 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:00.480 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:40:00.488 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:40:00.489 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863200 2023-10-21T04:40:00.489 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:00.489 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:40:00.490 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:40:00.490 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:40:10.491 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:40:10.491 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:10.493 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:10.495 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:11.396 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:40:11.396 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:40:11.398 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863211 2023-10-21T04:40:11.398 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:11.398 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:40:11.398 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:40:11.398 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:40:21.405 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:40:21.412 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:21.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:21.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:22.529 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:40:22.529 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:40:22.530 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863222 2023-10-21T04:40:22.530 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:22.530 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:40:22.531 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:40:22.531 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:40:32.536 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:40:32.539 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:32.540 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:32.547 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:33.600 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:40:33.610 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:40:33.612 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863233 2023-10-21T04:40:33.612 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:33.612 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:40:33.612 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:40:33.612 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:40:43.628 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:40:43.631 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:43.645 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:43.647 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:44.636 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:40:44.637 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:40:44.637 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863244 2023-10-21T04:40:44.638 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:44.638 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:40:44.638 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:40:44.638 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:40:54.639 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:40:54.639 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:54.640 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:54.647 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:55.532 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:40:55.536 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:40:55.536 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863255 2023-10-21T04:40:55.537 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:55.537 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:40:55.537 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:40:55.537 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:40:57.600 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:40:57.598+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:40:57.601 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:40:57.598+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:40:57.601 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:40:57.598+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:41:05.538 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:41:05.538 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:05.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:05.554 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:06.460 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:41:06.461 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:41:06.468 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863266 2023-10-21T04:41:06.468 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:06.468 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:41:06.469 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:41:06.469 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:41:15.084 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:41:15.085+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:41:15.639 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:15.637+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:41:15.640 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:15.637+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:41:16.471 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:16.475 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:41:16.478 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:16.479 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:17.413 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:41:17.414 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:41:17.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863277 2023-10-21T04:41:17.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:17.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:41:17.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:41:17.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:41:21.837 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:21.837+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:41:21.837 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:21.837+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:41:21.838 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:21.837+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:41:21.838 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:21.837+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T04:41:21.838 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:21.837+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:41:21.838 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:21.837+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:41:21.838 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:21.837+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:41:21.839 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:21.837+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:41:21.839 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:21.837+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:41:21.839 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:21.837+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:41:27.416 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:41:27.416 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:27.419 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:27.419 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:28.398 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:41:28.403 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:41:28.409 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863288 2023-10-21T04:41:28.409 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:28.409 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:41:28.410 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:41:28.410 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:41:38.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:41:38.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:38.415 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:38.419 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:39.492 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:41:39.492 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:41:39.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863299 2023-10-21T04:41:39.494 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:39.494 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:41:39.494 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:41:39.495 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:41:49.495 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:49.501 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:49.501 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:41:49.517 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:50.491 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:41:50.491 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:41:50.492 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863310 2023-10-21T04:41:50.492 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:50.492 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:41:50.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:41:50.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:41:55.769 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:55.764+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:41:55.770 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:55.764+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:41:55.770 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:41:55.764+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:42:00.494 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:42:00.501 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:00.501 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:00.503 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:42:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:42:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863321 2023-10-21T04:42:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:42:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:42:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:42:11.338 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:11.339 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:42:11.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:11.350 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:12.260 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:42:12.262 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:42:12.263 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863332 2023-10-21T04:42:12.263 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:12.263 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:42:12.263 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:42:12.264 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:42:17.753 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:42:17.751+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:42:17.753 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:42:17.751+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:42:22.272 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:22.273 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:22.274 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:42:22.276 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:23.172 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:42:23.176 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:42:23.177 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863343 2023-10-21T04:42:23.177 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:23.177 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:42:23.177 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:42:23.178 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:42:33.179 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:42:33.179 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:33.179 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:33.194 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:34.117 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:42:34.118 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:42:34.128 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863354 2023-10-21T04:42:34.128 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:34.128 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:42:34.129 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:42:34.129 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:42:44.130 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:42:44.131 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:44.134 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:44.140 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:45.040 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:42:45.040 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:42:45.041 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863365 2023-10-21T04:42:45.041 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:45.041 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:42:45.041 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:42:45.041 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:42:51.171 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:42:51.166+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:42:51.172 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:42:51.170+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:42:51.172 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:42:51.170+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:42:51.173 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:42:51.170+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:42:51.173 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:42:51.170+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:42:51.173 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:42:51.170+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:42:55.043 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:55.043 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:55.045 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:42:55.067 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:56.012 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:42:56.012 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:42:56.013 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863376 2023-10-21T04:42:56.013 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:56.014 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:42:56.014 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:42:56.014 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:43:06.015 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:43:06.020 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:06.021 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:06.022 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:06.894 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:43:06.895 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:43:06.896 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863386 2023-10-21T04:43:06.896 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:06.896 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:43:06.896 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:43:06.897 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:43:10.390 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:10.390+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:43:10.394 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:10.394+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:43:10.394 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:10.394+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:43:10.395 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:10.394+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:43:10.395 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:10.394+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:43:10.395 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:10.394+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:43:10.395 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:10.394+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:43:10.395 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:10.394+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:43:10.396 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:10.394+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:43:16.904 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:43:16.905 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:16.905 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:16.905 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:17.535 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:43:17.533+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T04:43:17.536 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:43:17.533+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130c30 2023-10-21T04:43:17.536 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:43:17.533+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:43:17.536 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:43:17.533+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:43:17.536 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:43:17.533+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:43:17.876 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:43:17.876 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:43:17.877 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863397 2023-10-21T04:43:17.877 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:17.877 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:43:17.877 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:43:17.877 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:43:27.879 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:43:27.879 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:27.891 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:27.897 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:28.730 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:43:28.731 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:43:28.731 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863408 2023-10-21T04:43:28.732 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:28.732 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:43:28.732 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:43:28.733 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:43:35.389 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:35.385+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:43:35.402 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:35.401+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:43:35.403 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:35.401+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:43:35.403 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:35.401+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:43:35.403 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:35.401+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:43:35.403 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:35.401+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:43:35.403 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:35.401+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:43:35.404 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:35.401+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:43:35.404 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:35.401+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:43:38.732 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:43:38.732 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:38.736 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:38.767 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:39.705 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:43:39.705 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:43:39.705 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863419 2023-10-21T04:43:39.706 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:39.706 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:43:39.706 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:43:39.706 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:43:49.707 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:49.708 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:43:49.708 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:49.722 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:50.608 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:43:50.608 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:43:50.609 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863430 2023-10-21T04:43:50.609 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:50.609 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:43:50.609 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:43:50.610 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:43:57.951 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:57.932+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:43:57.951 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:57.932+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:43:57.952 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:57.932+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:43:57.952 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:57.932+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:43:57.952 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:57.932+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:43:57.952 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:57.932+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:43:57.952 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:43:57.932+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:44:00.611 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:44:00.611 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:00.612 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:00.627 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:01.552 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:44:01.555 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:44:01.557 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863441 2023-10-21T04:44:01.557 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:01.557 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:44:01.557 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:44:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:44:11.565 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:44:11.565 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:11.565 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:11.570 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:12.472 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:44:12.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:44:12.473 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863452 2023-10-21T04:44:12.473 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:12.473 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:44:12.473 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:44:12.474 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:44:22.475 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:44:22.475 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:22.475 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:22.479 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:23.337 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:44:23.339 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:44:23.340 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863463 2023-10-21T04:44:23.340 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:23.341 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:44:23.341 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:44:23.341 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:44:33.343 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:33.343 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:33.345 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:33.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:44:34.210 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:44:34.211 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:44:34.211 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863474 2023-10-21T04:44:34.211 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:34.212 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:44:34.212 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:44:34.212 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:44:42.945 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:44:42.939+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:44:42.946 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:44:42.943+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:44:42.946 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:44:42.943+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:44:42.947 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:44:42.943+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:44:42.947 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:44:42.943+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:44:42.947 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:44:42.943+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:44:42.947 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:44:42.943+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:44:42.947 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:44:42.943+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:44:42.948 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:44:42.943+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:44:44.213 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:44:44.213 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:44.214 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:44.214 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:45.135 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:44:45.135 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:44:45.136 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863485 2023-10-21T04:44:45.136 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:45.136 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:44:45.136 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:44:45.136 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:44:55.143 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:44:55.143 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:55.143 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:55.149 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:56.090 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:44:56.097 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:44:56.098 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863496 2023-10-21T04:44:56.099 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:56.099 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:44:56.099 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:44:56.099 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:45:06.101 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:45:06.102 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:06.106 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:06.114 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:07.018 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:45:07.018 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:45:07.019 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863507 2023-10-21T04:45:07.019 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:07.019 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:45:07.019 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:45:07.020 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:45:17.020 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:45:17.021 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:17.022 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:17.025 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:17.887 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:45:17.887 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:45:17.888 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863517 2023-10-21T04:45:17.888 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:17.888 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:45:17.889 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:45:17.889 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:45:27.890 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:45:27.890 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:27.893 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:27.904 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:28.862 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:45:28.862 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:45:28.862 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863528 2023-10-21T04:45:28.863 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:28.863 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:45:28.863 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:45:28.863 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:45:38.863 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:45:38.864 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:38.867 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:38.873 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:39.727 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:45:39.733 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:45:39.734 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863539 2023-10-21T04:45:39.734 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:39.735 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:45:39.735 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:45:39.735 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:45:49.742 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:45:49.743 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:49.743 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:49.756 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:50.613 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:45:50.613 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:45:50.614 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863550 2023-10-21T04:45:50.614 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:50.614 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:45:50.614 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:45:50.615 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:46:00.620 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:46:00.624 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:00.625 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:00.643 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:46:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:46:01.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863561 2023-10-21T04:46:01.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:01.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:46:01.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:46:01.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:46:05.026 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:46:05.020+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:46:05.026 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:46:05.020+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:46:05.027 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:46:05.020+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:46:11.525 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:46:11.525 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:11.525 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:11.526 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:12.426 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:46:12.426 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:46:12.427 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863572 2023-10-21T04:46:12.427 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:12.427 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:46:12.427 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:46:12.428 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:46:22.428 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:22.429 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:46:22.431 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:22.442 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:23.369 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:46:23.369 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:46:23.370 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863583 2023-10-21T04:46:23.370 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:23.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:46:23.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:46:23.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:46:23.597 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:46:23.592+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:46:23.597 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:46:23.592+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:46:23.598 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:46:23.592+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:46:33.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:46:33.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:33.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:33.385 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:34.327 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:46:34.327 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:46:34.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863594 2023-10-21T04:46:34.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:34.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:46:34.329 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:46:34.329 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:46:44.333 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:46:44.333 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:44.334 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:44.349 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:45.342 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:46:45.342 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:46:45.343 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863605 2023-10-21T04:46:45.343 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:45.343 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:46:45.343 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:46:45.344 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:46:46.272 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:46:46.267+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:46:46.289 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:46:46.283+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:46:46.289 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:46:46.283+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:46:46.290 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:46:46.283+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:46:46.290 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:46:46.283+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:46:46.290 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:46:46.283+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:46:46.290 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:46:46.283+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:46:55.350 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:55.351 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:55.352 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:55.352 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:46:56.294 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:46:56.294 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:46:56.295 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863616 2023-10-21T04:46:56.295 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:56.295 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:46:56.296 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:46:56.296 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:47:06.296 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:06.297 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:06.300 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:47:06.314 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:07.348 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:47:07.348 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:47:07.349 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863627 2023-10-21T04:47:07.349 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:07.349 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:47:07.349 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:47:07.349 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:47:17.350 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:47:17.351 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:17.351 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:17.352 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:18.369 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:47:18.370 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:47:18.370 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863638 2023-10-21T04:47:18.370 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:18.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:47:18.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:47:18.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:47:24.932 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:47:24.930+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:47:24.933 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:47:24.930+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:47:24.933 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:47:24.930+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:47:28.380 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:28.380 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:28.384 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:47:28.384 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:29.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:47:29.329 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:47:29.329 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863649 2023-10-21T04:47:29.330 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:29.330 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:47:29.330 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:47:29.330 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:47:34.686 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:34.682+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:47:34.686 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:34.682+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:47:39.335 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:47:39.341 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:39.350 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:39.352 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:40.382 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:47:40.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:47:40.384 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863660 2023-10-21T04:47:40.384 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:40.384 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:47:40.385 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:47:40.385 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:47:43.038 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.029+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:47:43.039 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.033+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:47:43.039 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.033+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:47:43.040 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.033+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:47:43.040 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.033+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:47:43.040 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.033+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:47:43.922 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.917+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:47:43.926 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.925+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:47:43.926 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.925+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:47:43.926 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.925+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:47:43.927 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.925+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:47:43.927 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.925+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:47:43.927 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:47:43.925+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:47:44.705 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:47:44.701+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:47:44.706 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:47:44.701+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:47:50.388 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:47:50.389 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:50.390 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:50.426 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:51.416 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:47:51.417 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:47:51.418 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863671 2023-10-21T04:47:51.418 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:51.418 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:47:51.418 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:47:51.419 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:48:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:48:02.270 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:48:02.265+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:48:02.289 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:48:02.289 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:48:02.290 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863682 2023-10-21T04:48:02.290 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:02.290 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:48:02.290 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:48:02.291 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:48:12.292 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:12.293 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:12.297 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:48:12.309 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:13.234 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:48:13.235 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:48:13.235 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863693 2023-10-21T04:48:13.235 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:13.235 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:48:13.236 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:48:13.236 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:48:14.254 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:48:14.253+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:48:23.251 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:48:23.259 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:23.262 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:23.266 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:24.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:48:24.328 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:48:24.329 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863704 2023-10-21T04:48:24.329 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:24.329 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:48:24.329 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:48:24.330 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:48:24.750 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:48:24.748+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:48:34.331 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:48:34.332 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:34.333 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:34.335 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:35.157 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:48:35.157 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:48:35.158 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863715 2023-10-21T04:48:35.158 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:35.158 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:48:35.158 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:48:35.159 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:48:36.994 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:48:36.992+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:48:36.995 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:48:36.992+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:48:45.159 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:48:45.160 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:45.174 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:45.180 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:46.127 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:48:46.133 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:48:46.134 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863726 2023-10-21T04:48:46.134 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:46.135 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:48:46.135 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:48:46.135 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:48:56.136 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:48:56.136 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:56.156 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:56.156 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:57.097 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:48:57.098 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:48:57.098 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863737 2023-10-21T04:48:57.099 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:57.099 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:48:57.099 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:48:57.099 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:49:07.104 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:07.106 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:07.116 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:07.121 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:49:07.908 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:49:07.908 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:49:07.909 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863747 2023-10-21T04:49:07.909 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:07.909 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:49:07.910 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:49:07.910 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:49:17.911 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:49:17.911 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:17.912 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:17.916 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:18.868 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:49:18.868 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:49:18.868 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863758 2023-10-21T04:49:18.869 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:18.869 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:49:18.869 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:49:18.869 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:49:20.973 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:20.967+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:49:20.973 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:20.967+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:49:28.871 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:49:28.875 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:28.877 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:28.877 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:29.834 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:49:29.846 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:49:29.847 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863769 2023-10-21T04:49:29.847 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:29.847 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:49:29.848 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:49:29.848 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:49:39.027 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.022+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:49:39.027 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.022+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:49:39.027 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.022+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:49:39.480 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.478+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:49:39.481 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.478+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:49:39.481 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.478+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:49:39.481 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.478+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T04:49:39.481 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.478+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:49:39.482 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.478+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:49:39.482 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.478+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:49:39.482 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.478+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:49:39.482 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.478+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:49:39.482 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:39.478+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:49:39.851 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:39.851 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:39.859 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:49:39.870 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:40.775 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:49:40.806 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:49:40.807 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863780 2023-10-21T04:49:40.807 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:40.807 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:49:40.808 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:49:40.808 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:49:40.930 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:49:40.926+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:49:50.815 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:49:50.816 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:50.817 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:50.841 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:51.708 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:49:51.709 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:49:51.710 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863791 2023-10-21T04:49:51.710 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:51.710 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:49:51.710 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:49:51.711 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:50:01.716 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:01.721 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:50:01.722 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:01.733 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:02.623 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:50:02.623 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:50:02.624 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863802 2023-10-21T04:50:02.624 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:02.624 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:50:02.624 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:50:02.625 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:50:11.342 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:50:11.342+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:50:11.344 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:50:11.342+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:50:11.344 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:50:11.342+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:50:12.626 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:50:12.626 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:12.629 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:12.629 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:13.497 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:50:13.497 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:50:13.498 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863813 2023-10-21T04:50:13.498 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:13.498 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:50:13.499 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:50:13.499 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:50:23.505 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:23.507 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:50:23.508 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:23.516 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:24.480 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:50:24.480 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:50:24.483 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863824 2023-10-21T04:50:24.484 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:24.484 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:50:24.484 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:50:24.484 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:50:34.489 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:34.490 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:34.494 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:34.498 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:50:35.463 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:50:35.463 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:50:35.464 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863835 2023-10-21T04:50:35.464 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:35.464 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:50:35.465 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:50:35.465 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:50:43.211 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:50:43.201+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:50:45.471 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:50:45.471 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:45.471 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:45.487 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:46.370 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:50:46.370 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:50:46.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863846 2023-10-21T04:50:46.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:46.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:50:46.372 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:50:46.372 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:50:56.372 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:50:56.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:56.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:56.382 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:57.337 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:50:57.338 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:50:57.338 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863857 2023-10-21T04:50:57.338 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:57.339 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:50:57.339 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:50:57.339 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:51:07.340 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:07.340 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:07.342 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:51:07.342 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:08.191 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:51:08.192 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:51:08.194 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863868 2023-10-21T04:51:08.195 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:08.195 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:51:08.195 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:51:08.195 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:51:10.344 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:10.340+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:51:10.345 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:10.340+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:51:10.345 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:10.340+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:51:10.345 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:10.340+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:51:10.345 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:10.340+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:51:10.345 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:10.340+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:51:10.346 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:10.340+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:51:10.346 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:10.340+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:51:10.346 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:10.340+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:51:17.878 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:17.872+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:51:17.888 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:17.880+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:51:17.888 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:17.880+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:51:17.895 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:17.892+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:51:17.896 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:17.892+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:51:17.896 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:17.892+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:51:17.896 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:51:17.892+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:51:18.197 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:18.198 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:18.206 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:18.210 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:51:19.226 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:51:19.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:51:19.235 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863879 2023-10-21T04:51:19.235 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:19.236 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:51:19.236 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:51:19.236 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:51:19.899 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:51:19.892+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:51:19.899 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:51:19.892+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:51:29.240 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:29.240 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:29.241 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:51:29.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:30.241 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:51:30.241 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:51:30.242 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863890 2023-10-21T04:51:30.242 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:30.242 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:51:30.243 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:51:30.243 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:51:40.250 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:51:40.251 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:40.251 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:40.257 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:40.329 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:51:40.324+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:51:40.329 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:51:40.324+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:51:41.226 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:51:41.230 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:51:41.231 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863901 2023-10-21T04:51:41.231 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:41.232 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:51:41.232 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:51:41.232 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:51:50.391 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:51:50.387+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T04:51:50.391 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:51:50.387+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130c30 2023-10-21T04:51:50.392 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:51:50.387+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:51:50.392 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:51:50.387+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:51:50.392 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:51:50.387+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:51:51.238 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:51:51.241 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:51.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:51.247 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:52.130 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:51:52.131 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:51:52.132 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863912 2023-10-21T04:51:52.132 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:52.132 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:51:52.132 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:51:52.132 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:52:00.621 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:00.615+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:52:00.621 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:00.615+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:52:00.621 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:00.615+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:52:00.621 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:00.615+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T04:52:00.622 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:00.615+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:52:00.622 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:00.615+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T04:52:00.622 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:00.615+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:52:00.622 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:00.615+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:52:00.622 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:00.615+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:52:00.623 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:00.615+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:52:02.133 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:52:02.134 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:02.134 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:02.157 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:03.026 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:52:03.026 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:52:03.027 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863923 2023-10-21T04:52:03.027 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:03.027 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:52:03.028 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:52:03.028 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:52:13.030 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:52:13.030 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:13.030 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:13.035 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:14.031 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:52:14.031 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:52:14.032 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863934 2023-10-21T04:52:14.032 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:14.033 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:52:14.033 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:52:14.033 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:52:24.047 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:24.049 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:24.054 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:52:24.054 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:24.875 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:52:24.875 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:52:24.878 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863944 2023-10-21T04:52:24.879 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:24.879 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:52:24.879 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:52:24.879 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:52:27.510 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:27.495+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:52:27.514 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:27.507+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:52:27.514 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:27.507+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:52:27.514 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:52:27.507+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:52:34.881 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:34.884 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:34.884 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:52:34.891 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:35.814 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:52:35.814 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:52:35.815 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863955 2023-10-21T04:52:35.815 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:35.815 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:52:35.816 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:52:35.816 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:52:45.819 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:52:45.822 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:45.823 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:45.829 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:46.726 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:52:46.727 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:52:46.728 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863966 2023-10-21T04:52:46.728 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:46.728 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:52:46.729 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:52:46.729 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:52:53.586 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:52:53.578+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:52:53.586 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:52:53.578+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:52:56.739 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:52:56.740 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:56.752 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:56.755 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:57.809 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:52:57.809 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:52:57.810 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863977 2023-10-21T04:52:57.810 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:57.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:52:57.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:52:57.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:53:07.812 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:07.812 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:53:07.814 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:07.816 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:08.816 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:53:08.816 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:53:08.817 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863988 2023-10-21T04:53:08.817 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:08.817 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:53:08.817 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:53:08.817 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:53:18.819 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:18.820 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:53:18.820 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:18.823 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:19.829 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:53:19.830 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:53:19.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697863999 2023-10-21T04:53:19.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:19.830 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:53:19.831 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:53:19.831 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:53:29.834 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:29.834 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:53:29.834 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:29.834 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:30.106 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:53:30.097+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:53:30.121 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:53:30.109+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:53:30.121 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:53:30.109+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:53:30.122 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:53:30.109+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T04:53:30.122 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:53:30.109+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:53:30.122 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:53:30.109+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:53:30.122 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:53:30.109+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:53:30.123 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:53:30.109+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:53:30.748 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:53:30.748 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:53:30.749 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864010 2023-10-21T04:53:30.749 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:30.749 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:53:30.749 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:53:30.749 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:53:40.757 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:53:40.758 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:40.766 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:40.767 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:41.705 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:53:41.705 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:53:41.706 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864021 2023-10-21T04:53:41.706 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:41.706 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:53:41.707 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:53:41.707 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:53:51.710 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:51.715 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:51.715 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:53:51.725 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:52.761 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:53:52.763 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:53:52.764 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864032 2023-10-21T04:53:52.765 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:52.765 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:53:52.765 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:53:52.765 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:54:02.770 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:54:02.770 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:02.784 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:02.792 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:03.721 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:54:03.722 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:54:03.723 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864043 2023-10-21T04:54:03.723 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:03.723 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:54:03.723 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:54:03.724 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:54:13.724 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:54:13.725 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:13.725 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:13.728 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:14.596 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:54:14.596 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:54:14.597 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864054 2023-10-21T04:54:14.597 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:14.597 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:54:14.598 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:54:14.598 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:54:18.576 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:54:18.572+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:54:24.599 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:54:24.600 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:24.600 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:24.601 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:25.768 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:54:25.769 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:54:25.770 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864065 2023-10-21T04:54:25.770 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:25.770 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:54:25.770 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:54:25.771 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:54:35.775 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:54:35.776 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:35.778 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:35.787 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:36.645 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:54:36.646 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:54:36.647 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864076 2023-10-21T04:54:36.647 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:36.647 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:54:36.647 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:54:36.647 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:54:46.648 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:54:46.654 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:46.665 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:46.665 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:47.090 INFO:tasks.workunit.client.0.smithi119.stderr:2023-10-21T04:54:47.087+0000 7fd5948dd640 0 --2- 172.21.15.119:0/3971821904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd580003e20 0x7fd580004200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-21T04:54:47.790 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:54:47.791 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:54:47.791 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864087 2023-10-21T04:54:47.792 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:47.792 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:54:47.792 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:54:47.792 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:54:57.798 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:54:57.798 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:57.801 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:57.814 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:58.719 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:54:58.719 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:54:58.720 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864098 2023-10-21T04:54:58.720 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:58.720 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:54:58.720 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:54:58.720 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:55:08.721 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:55:08.722 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:08.723 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:08.725 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:09.600 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:55:09.600 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:55:09.601 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864109 2023-10-21T04:55:09.601 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:09.601 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:55:09.601 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:55:09.602 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:55:19.602 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:19.610 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:19.612 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:55:19.621 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:20.560 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:55:20.561 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:55:20.561 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864120 2023-10-21T04:55:20.561 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:20.561 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:55:20.562 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:55:20.562 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:55:30.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:30.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:30.572 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:30.586 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:55:31.492 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:55:31.492 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:55:31.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864131 2023-10-21T04:55:31.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:31.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:55:31.494 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:55:31.494 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:55:41.496 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:55:41.497 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:41.499 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:41.506 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:42.313 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:55:42.314 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:55:42.314 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864142 2023-10-21T04:55:42.315 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:42.315 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:55:42.315 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:55:42.315 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:55:52.320 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:52.320 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:55:52.322 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:52.323 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:53.303 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:55:53.304 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:55:53.304 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864153 2023-10-21T04:55:53.304 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:53.305 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:55:53.305 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:55:53.305 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:56:03.307 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:03.312 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:56:03.313 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:03.325 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:04.188 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:56:04.188 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:56:04.189 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864164 2023-10-21T04:56:04.189 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:04.189 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:56:04.189 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:56:04.190 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:56:14.206 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:56:14.207 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:14.208 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:14.215 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:15.053 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:56:15.054 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:56:15.054 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864175 2023-10-21T04:56:15.054 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:15.055 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:56:15.055 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:56:15.055 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:56:25.057 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:25.057 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:25.058 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:56:25.058 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:25.986 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:56:25.986 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:56:25.987 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864185 2023-10-21T04:56:25.987 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:25.987 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:56:25.987 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:56:25.988 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:56:35.990 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:35.991 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:35.991 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:56:36.006 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:36.892 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:56:36.892 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:56:36.893 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864196 2023-10-21T04:56:36.893 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:36.893 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:56:36.893 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:56:36.894 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:56:46.894 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:56:46.903 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:46.907 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:46.908 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:48.015 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:56:48.015 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:56:48.016 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864208 2023-10-21T04:56:48.016 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:48.016 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:56:48.016 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:56:48.017 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:56:55.888 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:56:55.876+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:56:55.888 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:56:55.876+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:56:55.889 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:56:55.876+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:56:58.017 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:56:58.018 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:58.019 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:58.047 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:58.874 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:56:58.874 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:56:58.874 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864218 2023-10-21T04:56:58.875 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:58.875 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:56:58.875 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:56:58.875 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:57:04.454 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:57:04.448+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:57:04.457 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:57:04.452+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:57:04.457 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:57:04.452+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:57:04.457 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:57:04.452+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:57:04.457 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:57:04.452+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:57:08.874 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:08.875 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:57:08.875 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:08.875 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:09.690 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:57:09.694 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:57:09.695 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864229 2023-10-21T04:57:09.695 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:09.695 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:57:09.695 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:57:09.696 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:57:19.704 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:19.704 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:19.712 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:19.723 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:57:20.610 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:57:20.610 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:57:20.611 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864240 2023-10-21T04:57:20.611 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:20.611 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:57:20.612 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:57:20.612 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:57:30.619 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:57:30.625 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:30.626 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:30.629 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:31.505 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:57:31.507 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:57:31.508 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864251 2023-10-21T04:57:31.508 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:31.508 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:57:31.509 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:57:31.509 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:57:41.523 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:57:41.523 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:41.526 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:41.532 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:42.413 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:57:42.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:57:42.414 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864262 2023-10-21T04:57:42.414 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:42.414 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:57:42.414 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:57:42.414 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:57:52.429 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:57:52.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:52.434 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:52.439 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:53.333 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:57:53.333 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:57:53.334 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864273 2023-10-21T04:57:53.334 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:53.334 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:57:53.335 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:57:53.335 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:57:59.226 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:57:59.219+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:57:59.226 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:57:59.219+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:58:03.336 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:58:03.336 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:03.342 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:03.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:04.319 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:58:04.325 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:58:04.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864284 2023-10-21T04:58:04.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:04.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:58:04.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:58:04.329 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:58:14.331 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:58:14.332 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:14.333 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:14.348 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:15.182 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:58:15.182 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:58:15.184 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864295 2023-10-21T04:58:15.185 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:15.185 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:58:15.185 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:58:15.185 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:58:25.187 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:58:25.188 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:25.190 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:25.191 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:26.173 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:58:26.173 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:58:26.174 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864306 2023-10-21T04:58:26.174 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:26.174 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:58:26.175 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:58:26.175 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:58:36.176 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:58:36.176 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:36.178 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:36.179 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:37.142 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:58:37.143 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:58:37.143 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864317 2023-10-21T04:58:37.144 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:37.144 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:58:37.144 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:58:37.144 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:58:42.358 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:58:42.354+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T04:58:42.358 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:58:42.354+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T04:58:42.359 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:58:42.354+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T04:58:42.359 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T04:58:42.354+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T04:58:47.145 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:47.147 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:47.148 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:47.176 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:58:47.183 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:58:47.182+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T04:58:47.189 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:58:47.186+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T04:58:47.189 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:58:47.186+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:58:48.108 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:58:48.108 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:58:48.109 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864328 2023-10-21T04:58:48.109 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:48.109 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:58:48.110 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:58:48.110 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:58:49.784 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:58:49.781+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T04:58:49.800 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:58:49.797+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T04:58:49.801 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:58:49.797+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T04:58:49.801 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:58:49.797+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T04:58:49.801 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T04:58:49.797+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T04:58:58.113 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:58:58.114 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:58.118 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:58.122 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:59.106 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:58:59.107 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:58:59.107 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864339 2023-10-21T04:58:59.108 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:59.108 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:58:59.108 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:58:59.108 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:59:09.109 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:09.111 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:09.111 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:09.113 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:59:09.986 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:59:09.987 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:59:09.987 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864349 2023-10-21T04:59:09.988 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:09.988 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:59:09.988 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:59:09.988 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:59:19.990 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:19.993 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:20.018 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:20.018 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:59:20.996 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:59:20.996 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:59:20.996 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864360 2023-10-21T04:59:20.997 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:20.997 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:59:20.997 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:59:20.997 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:59:31.012 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:31.020 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:59:31.020 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:31.034 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:31.981 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:59:31.983 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:59:31.987 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864371 2023-10-21T04:59:31.987 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:31.987 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:59:31.988 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:59:31.988 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:59:41.989 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:41.990 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:41.994 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:59:41.995 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:42.861 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:59:42.864 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:59:42.865 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864382 2023-10-21T04:59:42.866 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:42.866 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:59:42.866 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:59:42.866 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T04:59:52.887 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T04:59:52.887 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:52.888 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:52.895 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:53.757 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T04:59:53.758 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T04:59:53.758 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864393 2023-10-21T04:59:53.758 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:53.758 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T04:59:53.759 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T04:59:53.759 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:00:03.760 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:00:03.761 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:03.762 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:03.763 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:04.664 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:00:04.664 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:00:04.665 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864404 2023-10-21T05:00:04.665 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:04.666 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:00:04.666 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:00:04.666 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:00:14.670 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:00:14.672 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:14.675 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:14.678 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:15.481 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:00:15.481 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:00:15.482 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864415 2023-10-21T05:00:15.482 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:15.483 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:00:15.483 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:00:15.483 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:00:18.490 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:00:18.483+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:00:18.490 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:00:18.483+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:00:18.491 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:00:18.483+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:00:18.491 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:00:18.483+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:00:18.491 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:00:18.483+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:00:18.491 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:00:18.483+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:00:18.491 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:00:18.483+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:00:25.483 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:00:25.484 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:25.488 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:25.513 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:26.500 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:00:26.500 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:00:26.501 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864426 2023-10-21T05:00:26.501 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:26.501 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:00:26.501 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:00:26.501 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:00:36.510 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:00:36.516 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:36.521 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:36.555 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:37.470 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:00:37.470 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:00:37.471 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864437 2023-10-21T05:00:37.471 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:37.472 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:00:37.472 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:00:37.472 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:00:37.922 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:00:37.915+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T05:00:37.922 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:00:37.915+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130c30 2023-10-21T05:00:37.923 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:00:37.915+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T05:00:37.923 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:00:37.915+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T05:00:37.923 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:00:37.915+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:00:47.487 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:47.489 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:47.489 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:00:47.512 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:48.367 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:00:48.367 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:00:48.368 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864448 2023-10-21T05:00:48.368 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:48.368 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:00:48.369 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:00:48.369 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:00:58.370 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:58.370 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:58.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:58.378 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:00:59.369 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:00:59.370 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:00:59.370 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864459 2023-10-21T05:00:59.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:59.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:00:59.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:00:59.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:01:09.372 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:09.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:01:09.374 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:09.378 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:10.253 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:01:10.253 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:01:10.253 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864470 2023-10-21T05:01:10.254 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:10.254 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:01:10.254 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:01:10.254 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:01:11.182 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:01:11.178+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:01:16.646 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:01:16.642+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:01:16.646 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:01:16.642+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:01:20.256 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:20.256 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:20.257 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:01:20.262 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:21.224 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:01:21.225 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:01:21.227 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864481 2023-10-21T05:01:21.227 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:21.227 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:01:21.228 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:01:21.228 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:01:28.911 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:01:28.906+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:01:28.912 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:01:28.910+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:01:28.921 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:01:28.918+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:01:28.934 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:01:28.930+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:01:28.935 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:01:28.930+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T05:01:28.935 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:01:28.930+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:01:28.935 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:01:28.930+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:01:28.935 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:01:28.930+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:01:28.941 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:01:28.938+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:01:31.243 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:31.243 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:01:31.247 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:31.249 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:32.110 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:01:32.110 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:01:32.111 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864492 2023-10-21T05:01:32.111 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:32.111 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:01:32.112 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:01:32.112 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:01:38.794 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:01:38.789+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T05:01:38.794 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:01:38.789+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:01:42.127 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:01:42.129 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:42.135 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:42.138 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:42.968 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:01:42.975 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:01:42.983 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864502 2023-10-21T05:01:42.984 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:42.984 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:01:42.984 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:01:42.984 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:01:52.990 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:01:52.991 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:52.997 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:53.009 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:53.836 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:01:53.836 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:01:53.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864513 2023-10-21T05:01:53.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:53.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:01:53.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:01:53.838 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:02:03.838 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:02:03.839 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:03.842 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:03.851 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:04.726 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:02:04.738 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:02:04.743 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864524 2023-10-21T05:02:04.743 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:04.743 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:02:04.743 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:02:04.743 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:02:14.747 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:02:14.749 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:14.758 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:14.758 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:15.594 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:02:15.595 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:02:15.595 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864535 2023-10-21T05:02:15.596 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:15.596 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:02:15.596 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:02:15.596 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:02:25.598 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:25.598 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:25.607 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:25.615 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:02:26.450 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:02:26.450 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:02:26.451 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864546 2023-10-21T05:02:26.451 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:26.451 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:02:26.451 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:02:26.452 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:02:26.815 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:26.804+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:02:26.816 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:26.804+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:02:31.731 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:31.728+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:02:31.743 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:31.740+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:02:31.743 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:31.740+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:02:31.743 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:31.740+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:02:31.743 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:31.740+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:02:36.259 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:36.256+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:02:36.260 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:36.256+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:02:36.260 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:36.256+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:02:36.460 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:36.462 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:36.465 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:36.489 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:02:37.367 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:02:37.369 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:02:37.370 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864557 2023-10-21T05:02:37.370 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:37.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:02:37.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:02:37.371 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:02:40.444 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:40.440+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:02:40.449 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:40.444+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:02:40.450 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:40.444+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:02:40.454 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:40.452+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:02:40.455 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:40.452+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T05:02:40.455 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:40.452+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:02:40.455 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:40.452+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:02:40.455 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:40.452+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:02:40.456 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:02:40.452+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:02:47.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:02:47.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:47.374 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:47.378 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:48.278 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:02:48.278 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:02:48.279 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864568 2023-10-21T05:02:48.280 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:48.280 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:02:48.280 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:02:48.281 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:02:58.280 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:02:58.283 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:58.283 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:58.292 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:59.125 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:02:59.125 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:02:59.126 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864579 2023-10-21T05:02:59.126 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:59.126 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:02:59.126 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:02:59.127 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:03:09.142 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:09.144 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:03:09.147 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:09.149 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:10.033 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:03:10.034 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:03:10.034 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864590 2023-10-21T05:03:10.034 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:10.035 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:03:10.035 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:03:10.035 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:03:20.036 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:20.038 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:20.041 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:03:20.045 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:20.947 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:03:20.948 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:03:20.948 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864600 2023-10-21T05:03:20.948 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:20.949 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:03:20.949 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:03:20.949 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:03:29.464 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:03:29.459+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:03:29.465 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:03:29.459+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:03:29.465 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:03:29.459+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:03:29.465 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:03:29.459+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T05:03:29.465 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:03:29.459+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:03:29.466 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:03:29.459+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T05:03:29.466 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:03:29.459+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:03:29.466 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:03:29.459+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:03:29.466 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:03:29.459+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:03:29.466 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:03:29.459+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:03:30.950 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:03:30.951 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:30.951 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:30.962 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:31.907 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:03:31.907 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:03:31.908 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864611 2023-10-21T05:03:31.908 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:31.909 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:03:31.909 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:03:31.909 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:03:41.918 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:41.918 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:41.920 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:41.935 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:03:42.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:03:42.837 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:03:42.837 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864622 2023-10-21T05:03:42.838 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:42.838 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:03:42.838 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:03:42.838 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:03:49.517 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:03:49.514+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:03:52.846 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:52.849 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:52.853 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:03:52.856 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:53.816 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:03:53.816 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:03:53.818 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864633 2023-10-21T05:03:53.818 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:53.818 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:03:53.818 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:03:53.818 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:04:03.819 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:04:03.820 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:04:03.828 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:04:03.835 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:04:04.701 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:04:04.702 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:04:04.702 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864644 2023-10-21T05:04:04.702 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:04:04.702 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:04:04.703 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:04:04.703 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:04:06.175 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:06.166+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:04:06.186 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:06.174+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:04:06.186 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:06.174+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:04:06.570 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:06.566+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:04:06.570 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:06.566+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:04:06.570 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:06.566+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:04:14.703 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:04:14.706 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:04:14.710 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:04:14.721 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:04:15.595 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:04:15.596 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:04:15.596 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864655 2023-10-21T05:04:15.596 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:04:15.597 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:04:15.597 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:04:15.597 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:04:25.598 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:04:25.599 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:04:25.601 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:04:25.611 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:04:26.532 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:26.530+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:04:26.533 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:26.530+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:04:26.533 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:26.530+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:04:26.538 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:26.534+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:04:26.578 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:04:26.585 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:04:26.587 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864666 2023-10-21T05:04:26.587 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:04:26.587 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:04:26.588 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:04:26.588 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:04:30.835 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:04:30.829+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T05:04:30.835 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:04:30.829+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:04:36.593 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:04:36.594 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:04:36.597 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:04:36.598 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:04:37.555 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:04:37.556 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:04:37.557 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864677 2023-10-21T05:04:37.557 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:04:37.557 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:04:37.557 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:04:37.557 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:04:37.765 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:37.761+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:04:37.776 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:37.773+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:04:37.776 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:37.773+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:04:37.777 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:37.773+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:04:37.777 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:37.773+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T05:04:37.777 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:37.773+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:04:37.777 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:37.773+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:04:37.777 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:37.773+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:04:37.778 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:04:37.773+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:04:39.710 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:04:39.705+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T05:04:39.710 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:04:39.705+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T05:04:39.710 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:04:39.705+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:04:47.559 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:04:47.560 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:04:47.562 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:04:47.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:04:48.431 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:04:48.434 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:04:48.435 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864688 2023-10-21T05:04:48.436 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:04:48.436 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:04:48.436 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:04:48.436 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:04:58.446 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:04:58.447 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:04:58.447 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:04:58.447 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:04:59.394 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:04:59.394 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:04:59.395 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864699 2023-10-21T05:04:59.395 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:04:59.395 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:04:59.396 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:04:59.396 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:05:02.646 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:02.641+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:05:02.646 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:02.641+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:05:02.647 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:02.641+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:05:09.396 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:05:09.397 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:05:09.397 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:05:09.406 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:05:10.387 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:05:10.388 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:05:10.388 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864710 2023-10-21T05:05:10.388 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:05:10.389 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:05:10.389 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:05:10.389 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:05:20.390 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:05:20.393 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:05:20.399 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:05:20.407 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:05:21.402 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:05:21.406 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:05:21.407 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864721 2023-10-21T05:05:21.407 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:05:21.408 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:05:21.408 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:05:21.408 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:05:26.273 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:26.264+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:05:26.274 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:26.264+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:05:28.889 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:28.884+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:05:31.410 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:05:31.410 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:05:31.418 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:05:31.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:05:32.315 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:05:32.315 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:05:32.316 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864732 2023-10-21T05:05:32.316 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:05:32.316 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:05:32.316 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:05:32.317 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:05:34.492 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:34.488+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:05:34.493 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:34.488+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:05:34.493 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:34.488+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:05:34.493 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:34.488+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:05:34.493 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:34.488+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:05:34.494 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:34.488+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:05:35.716 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:35.712+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:05:35.742 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:35.736+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:05:35.743 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:35.736+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:05:35.743 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:05:35.736+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:05:42.318 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:05:42.318 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:05:42.318 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:05:42.318 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:05:43.182 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:05:43.184 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:05:43.185 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864743 2023-10-21T05:05:43.185 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:05:43.185 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:05:43.185 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:05:43.185 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:05:53.201 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:05:53.201 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:05:53.204 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:05:53.220 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:05:54.059 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:05:54.060 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:05:54.060 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864754 2023-10-21T05:05:54.060 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:05:54.061 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:05:54.061 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:05:54.061 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:06:04.072 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:06:04.078 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:06:04.099 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:06:04.109 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:06:04.944 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:06:04.945 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:06:04.953 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864764 2023-10-21T05:06:04.953 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:06:04.953 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:06:04.953 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:06:04.954 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:06:14.957 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:06:14.957 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:06:14.969 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:06:14.970 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:06:15.123 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:15.119+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:06:15.134 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:15.131+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:06:15.135 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:15.131+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:06:15.135 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:15.131+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:06:15.148 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:15.143+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:06:15.931 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:06:15.931 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:06:15.932 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864775 2023-10-21T05:06:15.932 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:06:15.932 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:06:15.932 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:06:15.932 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:06:18.161 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:18.155+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:06:18.162 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:18.155+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:06:18.162 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:18.155+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:06:18.711 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:18.707+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:06:18.711 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:18.707+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:06:18.711 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:18.707+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:06:18.723 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:18.719+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:06:22.017 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:22.011+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:06:22.028 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:22.023+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:06:22.028 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:22.023+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:06:22.028 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:22.023+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:06:25.943 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:06:25.948 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:06:25.948 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:06:25.950 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:06:26.866 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:06:26.866 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:06:26.867 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864786 2023-10-21T05:06:26.867 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:06:26.867 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:06:26.867 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:06:26.868 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:06:36.873 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:06:36.874 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:06:36.874 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:06:36.874 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:06:37.763 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:06:37.764 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:06:37.766 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864797 2023-10-21T05:06:37.766 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:06:37.766 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ -n 172.21.15.119:0/161644020 ]] 2023-10-21T05:06:37.767 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 172.21.15.119:0/161644020 != \1\7\2\.\2\1\.\1\5\.\1\1\9\:\0\/\1\6\1\6\4\4\0\2\0 ]] 2023-10-21T05:06:37.767 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:06:40.660 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:40.655+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:06:40.660 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:40.655+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:06:40.661 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:06:40.655+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:06:45.415 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:06:45.411+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T05:06:45.415 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:06:45.411+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130c30 2023-10-21T05:06:45.416 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:06:45.411+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T05:06:45.416 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:06:45.411+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T05:06:45.416 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:06:45.411+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:06:47.769 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.active_clients[]' 2023-10-21T05:06:47.770 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:06:47.774 INFO:tasks.workunit.client.0.smithi119.stderr:++ ceph mgr dump 2023-10-21T05:06:47.775 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:06:48.699 INFO:tasks.workunit.client.0.smithi119.stderr:+ CLIENT_ADDR=172.21.15.119:0/161644020 2023-10-21T05:06:48.700 INFO:tasks.workunit.client.0.smithi119.stderr:++ date +%s 2023-10-21T05:06:48.700 INFO:tasks.workunit.client.0.smithi119.stderr:+ CURRENT_TIME=1697864808 2023-10-21T05:06:48.700 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:06:48.700 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i = 1 )) 2023-10-21T05:06:48.701 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:06:48.701 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:06:48.852 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:06:48.846+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:06:48.853 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:06:48.866 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:06:58.436 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:06:58.430+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T05:06:58.436 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:06:58.430+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:06:58.874 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:06:58.874 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:06:58.874 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:06:58.975 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:06:58.974+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:06:58.975 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:06:58.988 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:07:08.993 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:07:08.993 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:07:08.994 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:09.081 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:07:09.082+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:09.082 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:09.089 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:07:10.760 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:10.758+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:07:10.760 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:10.758+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:07:10.760 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:10.758+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:07:10.761 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:10.758+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T05:07:10.761 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:10.758+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:07:10.761 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:10.758+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T05:07:10.761 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:10.758+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:07:10.761 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:10.758+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:07:10.762 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:10.758+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:07:10.762 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:10.758+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:07:16.222 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:16.222+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:07:16.222 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:16.222+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:07:16.223 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:16.222+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:07:16.231 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:16.230+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:07:19.095 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:07:19.095 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:07:19.095 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:19.197 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:07:19.194+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:19.198 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:19.216 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:07:29.219 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:07:29.220 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:07:29.220 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:29.308 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:07:29.306+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:29.309 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:29.314 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:07:39.315 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:07:39.316 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:07:39.316 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:39.404 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:07:39.401+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:39.405 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:39.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:07:44.818 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:44.809+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:07:44.819 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:44.809+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:07:44.819 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:44.809+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:07:49.420 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:07:49.420 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:07:49.421 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:49.504 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:07:49.501+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:49.505 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:49.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:07:56.683 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:56.681+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:07:56.683 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:56.681+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:07:56.684 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:56.681+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:07:56.684 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:56.681+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:07:56.684 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:07:56.681+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:07:59.527 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:07:59.528 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:07:59.528 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:59.649 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:07:59.645+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:59.649 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:59.667 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:08:08.228 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:08:08.224+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:08:08.229 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:08:08.224+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T05:08:08.229 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:08:08.224+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:08:08.229 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:08:08.224+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:08:08.229 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:08:08.224+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:08:08.229 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:08:08.224+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:08:09.675 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:08:09.675 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:08:09.676 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:08:09.791 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:08:09.788+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:08:09.792 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:08:09.807 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:08:19.809 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:08:19.809 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:08:19.809 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:08:19.891 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:08:19.888+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:08:19.896 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:08:19.907 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:08:29.909 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:08:29.909 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:08:29.909 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:08:29.981 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:08:29.976+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:08:29.981 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:08:30.008 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:08:40.010 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:08:40.011 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:08:40.011 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:08:40.097 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:08:40.092+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:08:40.098 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:08:40.107 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:08:50.111 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:08:50.111 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:08:50.111 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:08:50.204 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:08:50.199+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:08:50.204 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:08:50.214 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:09:00.216 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:09:00.216 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:09:00.216 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:09:00.308 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:09:00.303+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:09:00.308 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:09:00.327 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:09:10.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:09:10.328 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:09:10.329 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:09:10.612 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:09:10.607+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:09:10.612 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:09:10.617 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:09:20.625 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:09:20.625 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:09:20.625 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:09:20.729 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:09:20.723+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:09:20.729 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:09:20.746 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:09:30.755 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:09:30.756 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:09:30.756 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:09:30.890 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:09:30.890+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:09:30.890 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:09:30.906 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:09:40.907 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:09:40.908 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:09:40.908 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:09:40.995 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:09:40.994+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:09:40.996 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:09:40.999 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:09:51.001 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:09:51.002 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:09:51.002 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:09:51.121 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:09:51.118+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:09:51.121 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:09:51.140 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:09:52.626 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:09:52.622+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T05:09:52.627 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:09:52.622+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T05:09:52.627 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:09:52.622+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T05:09:52.627 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:09:52.622+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:10:01.146 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:10:01.147 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:10:01.147 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:10:01.238 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:10:01.234+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:10:01.239 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:10:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:10:11.267 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:10:11.267 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:10:11.268 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:10:11.345 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:10:11.341+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:10:11.346 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:10:11.350 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:10:21.351 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:10:21.351 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:10:21.351 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:10:21.427 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:10:21.425+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:10:21.428 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:10:21.441 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:10:31.445 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:10:31.445 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:10:31.445 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:10:31.540 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:10:31.537+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:10:31.541 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:10:31.553 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:10:41.555 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:10:41.555 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:10:41.556 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:10:41.629 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:10:41.624+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:10:41.629 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:10:41.642 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 10 2023-10-21T05:10:43.269 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:43.256+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:10:51.644 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i++ )) 2023-10-21T05:10:51.644 INFO:tasks.workunit.client.0.smithi119.stderr:+ (( i <= 24 )) 2023-10-21T05:10:51.644 INFO:tasks.workunit.client.0.smithi119.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-10-21T05:10:51.645 INFO:tasks.workunit.client.0.smithi119.stderr:+ grep 'every 2m' 2023-10-21T05:10:51.734 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-10-21T05:10:51.732+0000 7ffac79ce640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:10:51.735 INFO:tasks.workunit.client.0.smithi119.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:10:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T05:10:51.746 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-21T05:10:51.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-21T05:10:51.920 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:51.916+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:10:51.921 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:51.916+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:10:51.921 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:51.916+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:10:51.921 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:51.916+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T05:10:51.922 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:51.916+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:10:51.922 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:51.916+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T05:10:51.922 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:51.916+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:10:51.922 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:51.916+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:10:51.922 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:51.916+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:10:51.923 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:51.916+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:10:52.331 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-21T05:10:52.397 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3cb9a7fb51ac43a1ace7d64ab7b5f5b0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-21T05:10:52.400 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-21T05:10:52.447 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-21T05:10:52.456 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-21T05:10:52.456 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:10:53.139 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:10:53.139 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:10:53.173 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2054,"stamp":"2023-10-21T05:10:51.624951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27350326504,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1818646,"num_read_kb":7097501,"num_write":1798215,"num_write_kb":7191839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75877,"ondisk_log_size":75877,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18221212,"kb_used_data":14381892,"kb_used_omap":1457,"kb_used_meta":3837838,"kb_avail":264894308,"statfs":{"total":289910292480,"available":271251771392,"internally_reserved":0,"allocated":14727057408,"data_stored":16964064531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":3929946308},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,13,7,17,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16461824,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4507,"num_read_kb":17964,"num_write":4513,"num_write_kb":18052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001888"},"pg_stats":[{"pgid":"2.7","version":"79'8961","reported_seq":12364,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:46.032435+0000","last_change":"2023-10-21T04:05:01.187428+0000","last_active":"2023-10-21T05:10:46.032435+0000","last_peered":"2023-10-21T05:10:46.032435+0000","last_clean":"2023-10-21T05:10:46.032435+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:10:46.032435+0000","last_undegraded":"2023-10-21T05:10:46.032435+0000","last_fullsized":"2023-10-21T05:10:46.032435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":8961,"log_dups_size":0,"ondisk_log_size":8961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:46:46.175450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"59'31","reported_seq":976,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-21T05:10:06.644676+0000","last_change":"2023-10-21T04:05:01.187482+0000","last_active":"2023-10-21T05:10:06.644676+0000","last_peered":"2023-10-21T05:10:06.644676+0000","last_clean":"2023-10-21T05:10:06.644676+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:10:06.644676+0000","last_undegraded":"2023-10-21T05:10:06.644676+0000","last_fullsized":"2023-10-21T05:10:06.644676+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:11:54.174741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'14800","reported_seq":20315,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:46.046784+0000","last_change":"2023-10-21T04:05:01.187628+0000","last_active":"2023-10-21T05:10:46.046784+0000","last_peered":"2023-10-21T05:10:46.046784+0000","last_clean":"2023-10-21T05:10:46.046784+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:10:46.046784+0000","last_undegraded":"2023-10-21T05:10:46.046784+0000","last_fullsized":"2023-10-21T05:10:46.046784+0000","mapping_epoch":12,"log_start":"75'4700","ondisk_log_start":"75'4700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:50:24.700355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"79'3060","reported_seq":4781,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:44.707673+0000","last_change":"2023-10-21T04:05:01.188633+0000","last_active":"2023-10-21T05:10:44.707673+0000","last_peered":"2023-10-21T05:10:44.707673+0000","last_clean":"2023-10-21T05:10:44.707673+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:10:44.707673+0000","last_undegraded":"2023-10-21T05:10:44.707673+0000","last_fullsized":"2023-10-21T05:10:44.707673+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":3060,"log_dups_size":0,"ondisk_log_size":3060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:08:20.102163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"79'453137","reported_seq":904701,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:46.637076+0000","last_change":"2023-10-21T04:15:04.261265+0000","last_active":"2023-10-21T05:10:46.637076+0000","last_peered":"2023-10-21T05:10:46.637076+0000","last_clean":"2023-10-21T05:10:46.637076+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:10:46.637076+0000","last_undegraded":"2023-10-21T05:10:46.637076+0000","last_fullsized":"2023-10-21T05:10:46.637076+0000","mapping_epoch":15,"log_start":"78'443113","ondisk_log_start":"78'443113","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:39:42.118928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6874112000,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":453190,"num_read_kb":1782936,"num_write":451334,"num_write_kb":1805336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"79'14991","reported_seq":20727,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:45.573790+0000","last_change":"2023-10-21T04:05:01.187428+0000","last_active":"2023-10-21T05:10:45.573790+0000","last_peered":"2023-10-21T05:10:45.573790+0000","last_clean":"2023-10-21T05:10:45.573790+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:10:45.573790+0000","last_undegraded":"2023-10-21T05:10:45.573790+0000","last_fullsized":"2023-10-21T05:10:45.573790+0000","mapping_epoch":12,"log_start":"75'4900","ondisk_log_start":"75'4900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:02:28.019743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"79'448491","reported_seq":893267,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:46.855511+0000","last_change":"2023-10-21T04:15:04.363768+0000","last_active":"2023-10-21T05:10:46.855511+0000","last_peered":"2023-10-21T05:10:46.855511+0000","last_clean":"2023-10-21T05:10:46.855511+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:10:46.855511+0000","last_undegraded":"2023-10-21T05:10:46.855511+0000","last_fullsized":"2023-10-21T05:10:46.855511+0000","mapping_epoch":15,"log_start":"78'438416","ondisk_log_start":"78'438416","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:10:25.585633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6780858368,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":446354,"num_read_kb":1755324,"num_write":446736,"num_write_kb":1786944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'282","reported_seq":5981,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:44.810556+0000","last_change":"2023-10-21T04:05:01.189879+0000","last_active":"2023-10-21T05:10:44.810556+0000","last_peered":"2023-10-21T05:10:44.810556+0000","last_clean":"2023-10-21T05:10:44.810556+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:10:44.810556+0000","last_undegraded":"2023-10-21T05:10:44.810556+0000","last_fullsized":"2023-10-21T05:10:44.810556+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:41:43.561612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"79'465702","reported_seq":928185,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:46.635246+0000","last_change":"2023-10-21T04:15:04.256298+0000","last_active":"2023-10-21T05:10:46.635246+0000","last_peered":"2023-10-21T05:10:46.635246+0000","last_clean":"2023-10-21T05:10:46.635246+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:10:46.635246+0000","last_undegraded":"2023-10-21T05:10:46.635246+0000","last_fullsized":"2023-10-21T05:10:46.635246+0000","mapping_epoch":15,"log_start":"78'455623","ondisk_log_start":"78'455623","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:42:06.826330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7067488256,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":464177,"num_read_kb":1826140,"num_write":463838,"num_write_kb":1855352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'7","reported_seq":136,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-21T05:09:26.476861+0000","last_change":"2023-10-21T04:05:01.190036+0000","last_active":"2023-10-21T05:09:26.476861+0000","last_peered":"2023-10-21T05:09:26.476861+0000","last_clean":"2023-10-21T05:09:26.476861+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:09:26.476861+0000","last_undegraded":"2023-10-21T05:09:26.476861+0000","last_fullsized":"2023-10-21T05:09:26.476861+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:52:35.513076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"79'437247","reported_seq":872440,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:46.871380+0000","last_change":"2023-10-21T04:15:04.254514+0000","last_active":"2023-10-21T05:10:46.871380+0000","last_peered":"2023-10-21T05:10:46.871380+0000","last_clean":"2023-10-21T05:10:46.871380+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:10:46.871380+0000","last_undegraded":"2023-10-21T05:10:46.871380+0000","last_fullsized":"2023-10-21T05:10:46.871380+0000","mapping_epoch":15,"log_start":"78'427216","ondisk_log_start":"78'427216","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:55.129639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6627274771,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":436760,"num_read_kb":1717624,"num_write":435501,"num_write_kb":1741998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"79'3039","reported_seq":4295,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:44.707556+0000","last_change":"2023-10-21T04:05:01.186422+0000","last_active":"2023-10-21T05:10:44.707556+0000","last_peered":"2023-10-21T05:10:44.707556+0000","last_clean":"2023-10-21T05:10:44.707556+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:10:44.707556+0000","last_undegraded":"2023-10-21T05:10:44.707556+0000","last_fullsized":"2023-10-21T05:10:44.707556+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":3039,"log_dups_size":0,"ondisk_log_size":3039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:30:32.067265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":238,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-21T05:09:21.390470+0000","last_change":"2023-10-21T04:04:56.457261+0000","last_active":"2023-10-21T05:09:21.390470+0000","last_peered":"2023-10-21T05:09:21.390470+0000","last_clean":"2023-10-21T05:09:21.390470+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:09:21.390470+0000","last_undegraded":"2023-10-21T05:09:21.390470+0000","last_fullsized":"2023-10-21T05:09:21.390470+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:04:55.099596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:04:55.099596+0000","last_clean_scrub_stamp":"2023-10-21T04:04:55.099596+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:20:18.949598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27349733395,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1800481,"num_read_kb":7082024,"num_write":1797409,"num_write_kb":7189630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14724648960,"data_stored":16963317798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40209,"ondisk_log_size":40209,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":35571,"ondisk_log_size":35571,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739163,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4712320,"kb_used_data":3661944,"kb_used_omap":373,"kb_used_meta":1049994,"kb_avail":89659520,"statfs":{"total":96636764160,"available":91811348480,"internally_reserved":0,"allocated":3749830656,"data_stored":4321725761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1075194416},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,10,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.26200000000000001,"5min":0.39200000000000002,"15min":0.39400000000000002},"min":{"1min":0.17899999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.42199999999999999,"5min":4.8689999999999998,"15min":5.9009999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.40400000000000003,"15min":0.40300000000000002},"min":{"1min":0.19500000000000001,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":5.9349999999999996},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.66400000000000003,"15min":0.57299999999999995},"min":{"1min":0.20699999999999999,"5min":0.20100000000000001,"15min":0.17899999999999999},"max":{"1min":5.4100000000000001,"5min":14.318,"15min":14.318},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.70999999999999996,"15min":0.63200000000000001},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":5.4450000000000003,"5min":14.343,"15min":14.343},"last":0.35299999999999998}]}]},{"osd":2,"up_from":8,"seq":34359739162,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9109688,"kb_used_data":7189808,"kb_used_omap":724,"kb_used_meta":1919147,"kb_avail":85262152,"statfs":{"total":96636764160,"available":87308443648,"internally_reserved":0,"allocated":7362363392,"data_stored":8481244969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":1965207154},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.55800000000000005,"15min":0.42899999999999999},"min":{"1min":0.17899999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.8899999999999997,"5min":5.8899999999999997,"15min":6.383},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.29399999999999998,"5min":0.39000000000000001,"15min":0.45900000000000002},"min":{"1min":0.20300000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":4.9029999999999996,"15min":6.0289999999999999},"last":0.23899999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:09:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.67100000000000004,"15min":0.495},"min":{"1min":0.254,"5min":0.21299999999999999,"15min":0.19900000000000001},"max":{"1min":1.7909999999999999,"5min":26.620999999999999,"15min":26.620999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.72799999999999998,"15min":0.624},"min":{"1min":0.21099999999999999,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":1.8240000000000001,"5min":26.596,"15min":26.596},"last":0.315}]}]},{"osd":1,"up_from":8,"seq":34359739161,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4399204,"kb_used_data":3530140,"kb_used_omap":359,"kb_used_meta":868696,"kb_avail":89972636,"statfs":{"total":96636764160,"available":92131979264,"internally_reserved":0,"allocated":3614863360,"data_stored":4161093801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":889544738},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,12,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.38400000000000001,"15min":0.46999999999999997},"min":{"1min":0.186,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.51300000000000001,"5min":2.6320000000000001,"15min":13.757},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.0099999999999998,"5min":5.1459999999999999,"15min":13.183999999999999},"last":0.21099999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.55100000000000005},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.20599999999999999},"max":{"1min":1.782,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.36799999999999999,"15min":0.56399999999999995},"min":{"1min":0.23300000000000001,"5min":0.19800000000000001,"15min":0.193},"max":{"1min":1.462,"5min":2.6640000000000001,"15min":19.521999999999998},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3748548608,"data_stored":4321005568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3613650944,"data_stored":4160528403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7362449408,"data_stored":8481783827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:10:53.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:10:53.678 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:10:53.679 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:10:53.735 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2055,"stamp":"2023-10-21T05:10:53.625242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27358518504,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1820858,"num_read_kb":7106305,"num_write":1800465,"num_write_kb":7200839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75769,"ondisk_log_size":75769,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18292068,"kb_used_data":14400008,"kb_used_omap":1457,"kb_used_meta":3890574,"kb_avail":264823452,"statfs":{"total":289910292480,"available":271179214848,"internally_reserved":0,"allocated":14745608192,"data_stored":16982615315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":3983947972},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,7,5,15,14,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24653824,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6719,"num_read_kb":26768,"num_write":6763,"num_write_kb":27052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001881"},"pg_stats":[{"pgid":"2.7","version":"79'8970","reported_seq":12376,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.906406+0000","last_change":"2023-10-21T04:05:01.187428+0000","last_active":"2023-10-21T05:10:51.906406+0000","last_peered":"2023-10-21T05:10:51.906406+0000","last_clean":"2023-10-21T05:10:51.906406+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:10:51.906406+0000","last_undegraded":"2023-10-21T05:10:51.906406+0000","last_fullsized":"2023-10-21T05:10:51.906406+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":8970,"log_dups_size":0,"ondisk_log_size":8970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:46:46.175450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"59'31","reported_seq":976,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-21T05:10:06.644676+0000","last_change":"2023-10-21T04:05:01.187482+0000","last_active":"2023-10-21T05:10:06.644676+0000","last_peered":"2023-10-21T05:10:06.644676+0000","last_clean":"2023-10-21T05:10:06.644676+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:10:06.644676+0000","last_undegraded":"2023-10-21T05:10:06.644676+0000","last_fullsized":"2023-10-21T05:10:06.644676+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:11:54.174741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'14813","reported_seq":20332,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.908916+0000","last_change":"2023-10-21T04:05:01.187628+0000","last_active":"2023-10-21T05:10:51.908916+0000","last_peered":"2023-10-21T05:10:51.908916+0000","last_clean":"2023-10-21T05:10:51.908916+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:10:51.908916+0000","last_undegraded":"2023-10-21T05:10:51.908916+0000","last_fullsized":"2023-10-21T05:10:51.908916+0000","mapping_epoch":12,"log_start":"75'4800","ondisk_log_start":"75'4800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:50:24.700355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"79'3063","reported_seq":4785,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:50.514362+0000","last_change":"2023-10-21T04:05:01.188633+0000","last_active":"2023-10-21T05:10:50.514362+0000","last_peered":"2023-10-21T05:10:50.514362+0000","last_clean":"2023-10-21T05:10:50.514362+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:10:50.514362+0000","last_undegraded":"2023-10-21T05:10:50.514362+0000","last_fullsized":"2023-10-21T05:10:50.514362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":3063,"log_dups_size":0,"ondisk_log_size":3063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:08:20.102163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"79'453704","reported_seq":905815,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.623001+0000","last_change":"2023-10-21T04:15:04.261265+0000","last_active":"2023-10-21T05:10:51.623001+0000","last_peered":"2023-10-21T05:10:51.623001+0000","last_clean":"2023-10-21T05:10:51.623001+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:10:51.623001+0000","last_undegraded":"2023-10-21T05:10:51.623001+0000","last_fullsized":"2023-10-21T05:10:51.623001+0000","mapping_epoch":15,"log_start":"78'443613","ondisk_log_start":"78'443613","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:39:42.118928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6876164096,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":453736,"num_read_kb":1785116,"num_write":451901,"num_write_kb":1807604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"79'15006","reported_seq":20747,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:49.828049+0000","last_change":"2023-10-21T04:05:01.187428+0000","last_active":"2023-10-21T05:10:49.828049+0000","last_peered":"2023-10-21T05:10:49.828049+0000","last_clean":"2023-10-21T05:10:49.828049+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:10:49.828049+0000","last_undegraded":"2023-10-21T05:10:49.828049+0000","last_fullsized":"2023-10-21T05:10:49.828049+0000","mapping_epoch":12,"log_start":"75'5000","ondisk_log_start":"75'5000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:02:28.019743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"79'449052","reported_seq":894362,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.912691+0000","last_change":"2023-10-21T04:15:04.363768+0000","last_active":"2023-10-21T05:10:51.912691+0000","last_peered":"2023-10-21T05:10:51.912691+0000","last_clean":"2023-10-21T05:10:51.912691+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:10:51.912691+0000","last_undegraded":"2023-10-21T05:10:51.912691+0000","last_fullsized":"2023-10-21T05:10:51.912691+0000","mapping_epoch":15,"log_start":"78'439016","ondisk_log_start":"78'439016","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:10:25.585633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6782939136,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":446887,"num_read_kb":1757444,"num_write":447297,"num_write_kb":1789188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'282","reported_seq":5986,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:49.845481+0000","last_change":"2023-10-21T04:05:01.189879+0000","last_active":"2023-10-21T05:10:49.845481+0000","last_peered":"2023-10-21T05:10:49.845481+0000","last_clean":"2023-10-21T05:10:49.845481+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:10:49.845481+0000","last_undegraded":"2023-10-21T05:10:49.845481+0000","last_fullsized":"2023-10-21T05:10:49.845481+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:41:43.561612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"79'466266","reported_seq":929320,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.639798+0000","last_change":"2023-10-21T04:15:04.256298+0000","last_active":"2023-10-21T05:10:51.639798+0000","last_peered":"2023-10-21T05:10:51.639798+0000","last_clean":"2023-10-21T05:10:51.639798+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:10:51.639798+0000","last_undegraded":"2023-10-21T05:10:51.639798+0000","last_fullsized":"2023-10-21T05:10:51.639798+0000","mapping_epoch":15,"log_start":"78'456223","ondisk_log_start":"78'456223","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:42:06.826330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7069523968,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":464744,"num_read_kb":1828392,"num_write":464402,"num_write_kb":1857608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'7","reported_seq":136,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-21T05:09:26.476861+0000","last_change":"2023-10-21T04:05:01.190036+0000","last_active":"2023-10-21T05:09:26.476861+0000","last_peered":"2023-10-21T05:09:26.476861+0000","last_clean":"2023-10-21T05:09:26.476861+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:09:26.476861+0000","last_undegraded":"2023-10-21T05:09:26.476861+0000","last_fullsized":"2023-10-21T05:09:26.476861+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:52:35.513076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"79'437805","reported_seq":873565,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.913192+0000","last_change":"2023-10-21T04:15:04.254514+0000","last_active":"2023-10-21T05:10:51.913192+0000","last_peered":"2023-10-21T05:10:51.913192+0000","last_clean":"2023-10-21T05:10:51.913192+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:10:51.913192+0000","last_undegraded":"2023-10-21T05:10:51.913192+0000","last_fullsized":"2023-10-21T05:10:51.913192+0000","mapping_epoch":15,"log_start":"78'427716","ondisk_log_start":"78'427716","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:55.129639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6629298195,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":437326,"num_read_kb":1719876,"num_write":436059,"num_write_kb":1744230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"79'3041","reported_seq":4298,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:49.827350+0000","last_change":"2023-10-21T04:05:01.186422+0000","last_active":"2023-10-21T05:10:49.827350+0000","last_peered":"2023-10-21T05:10:49.827350+0000","last_clean":"2023-10-21T05:10:49.827350+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:10:49.827350+0000","last_undegraded":"2023-10-21T05:10:49.827350+0000","last_fullsized":"2023-10-21T05:10:49.827350+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":3041,"log_dups_size":0,"ondisk_log_size":3041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:30:32.067265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":238,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-21T05:09:21.390470+0000","last_change":"2023-10-21T04:04:56.457261+0000","last_active":"2023-10-21T05:09:21.390470+0000","last_peered":"2023-10-21T05:09:21.390470+0000","last_clean":"2023-10-21T05:09:21.390470+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:09:21.390470+0000","last_undegraded":"2023-10-21T05:09:21.390470+0000","last_fullsized":"2023-10-21T05:09:21.390470+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:04:55.099596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:04:55.099596+0000","last_clean_scrub_stamp":"2023-10-21T04:04:55.099596+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:20:18.949598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27357925395,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1802693,"num_read_kb":7090828,"num_write":1799659,"num_write_kb":7198630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14743121920,"data_stored":16981790758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40259,"ondisk_log_size":40259,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":35413,"ondisk_log_size":35413,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739164,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4721108,"kb_used_data":3666568,"kb_used_omap":373,"kb_used_meta":1054154,"kb_avail":89650732,"statfs":{"total":96636764160,"available":91802349568,"internally_reserved":0,"allocated":3754565632,"data_stored":4326460737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1079454256},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,2,1,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.26200000000000001,"5min":0.39200000000000002,"15min":0.39400000000000002},"min":{"1min":0.17899999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.42199999999999999,"5min":4.8689999999999998,"15min":5.9009999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.40400000000000003,"15min":0.40300000000000002},"min":{"1min":0.19500000000000001,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":5.9349999999999996},"last":0.34799999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.66400000000000003,"15min":0.57299999999999995},"min":{"1min":0.20699999999999999,"5min":0.20100000000000001,"15min":0.17899999999999999},"max":{"1min":5.4100000000000001,"5min":14.318,"15min":14.318},"last":0.20399999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.70999999999999996,"15min":0.63200000000000001},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":5.4450000000000003,"5min":14.343,"15min":14.343},"last":0.25}]}]},{"osd":2,"up_from":8,"seq":34359739163,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9163068,"kb_used_data":7198772,"kb_used_omap":724,"kb_used_meta":1963563,"kb_avail":85208772,"statfs":{"total":96636764160,"available":87253782528,"internally_reserved":0,"allocated":7371542528,"data_stored":8490424105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":2010689138},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.59999999999999998,"15min":0.435},"min":{"1min":0.17100000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":3.0699999999999998,"5min":5.8899999999999997,"15min":6.383},"last":0.316},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.33600000000000002,"15min":0.45600000000000002},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.56499999999999995,"5min":2.8610000000000002,"15min":6.0289999999999999},"last":0.221}]},{"osd":1,"last update":"Sat Oct 21 05:10:46 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.35999999999999999,"15min":0.496},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":0.56699999999999995,"5min":1.7909999999999999,"15min":26.620999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.41499999999999998,"15min":0.627},"min":{"1min":0.245,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":0.50900000000000001,"5min":2.9929999999999999,"15min":26.596},"last":0.29299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739162,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4407892,"kb_used_data":3534668,"kb_used_omap":359,"kb_used_meta":872856,"kb_avail":89963948,"statfs":{"total":96636764160,"available":92123082752,"internally_reserved":0,"allocated":3619500032,"data_stored":4165730473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":893804578},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,10,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.38400000000000001,"15min":0.46999999999999997},"min":{"1min":0.186,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.51300000000000001,"5min":2.6320000000000001,"15min":13.757},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.0099999999999998,"5min":5.1459999999999999,"15min":13.183999999999999},"last":0.46000000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.55100000000000005},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.20599999999999999},"max":{"1min":1.782,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.36799999999999999,"15min":0.56399999999999995},"min":{"1min":0.23300000000000001,"5min":0.19800000000000001,"15min":0.193},"max":{"1min":1.462,"5min":2.6640000000000001,"15min":19.521999999999998},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3753181184,"data_stored":4325638144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3618226176,"data_stored":4165103635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7371714560,"data_stored":8491048979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:10:53.737 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-21T05:10:53.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:10:54.300 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:10:54.300 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:10:54.346 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2055,"stamp":"2023-10-21T05:10:53.625242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27358518504,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1820858,"num_read_kb":7106305,"num_write":1800465,"num_write_kb":7200839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75769,"ondisk_log_size":75769,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18292068,"kb_used_data":14400008,"kb_used_omap":1457,"kb_used_meta":3890574,"kb_avail":264823452,"statfs":{"total":289910292480,"available":271179214848,"internally_reserved":0,"allocated":14745608192,"data_stored":16982615315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":3983947972},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,7,5,15,14,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24653824,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6719,"num_read_kb":26768,"num_write":6763,"num_write_kb":27052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001881"},"pg_stats":[{"pgid":"2.7","version":"79'8970","reported_seq":12376,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.906406+0000","last_change":"2023-10-21T04:05:01.187428+0000","last_active":"2023-10-21T05:10:51.906406+0000","last_peered":"2023-10-21T05:10:51.906406+0000","last_clean":"2023-10-21T05:10:51.906406+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:10:51.906406+0000","last_undegraded":"2023-10-21T05:10:51.906406+0000","last_fullsized":"2023-10-21T05:10:51.906406+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":8970,"log_dups_size":0,"ondisk_log_size":8970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:46:46.175450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"59'31","reported_seq":976,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-21T05:10:06.644676+0000","last_change":"2023-10-21T04:05:01.187482+0000","last_active":"2023-10-21T05:10:06.644676+0000","last_peered":"2023-10-21T05:10:06.644676+0000","last_clean":"2023-10-21T05:10:06.644676+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:10:06.644676+0000","last_undegraded":"2023-10-21T05:10:06.644676+0000","last_fullsized":"2023-10-21T05:10:06.644676+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:11:54.174741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'14813","reported_seq":20332,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.908916+0000","last_change":"2023-10-21T04:05:01.187628+0000","last_active":"2023-10-21T05:10:51.908916+0000","last_peered":"2023-10-21T05:10:51.908916+0000","last_clean":"2023-10-21T05:10:51.908916+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:10:51.908916+0000","last_undegraded":"2023-10-21T05:10:51.908916+0000","last_fullsized":"2023-10-21T05:10:51.908916+0000","mapping_epoch":12,"log_start":"75'4800","ondisk_log_start":"75'4800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:50:24.700355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"79'3063","reported_seq":4785,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:50.514362+0000","last_change":"2023-10-21T04:05:01.188633+0000","last_active":"2023-10-21T05:10:50.514362+0000","last_peered":"2023-10-21T05:10:50.514362+0000","last_clean":"2023-10-21T05:10:50.514362+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:10:50.514362+0000","last_undegraded":"2023-10-21T05:10:50.514362+0000","last_fullsized":"2023-10-21T05:10:50.514362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":3063,"log_dups_size":0,"ondisk_log_size":3063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:08:20.102163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"79'453704","reported_seq":905815,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.623001+0000","last_change":"2023-10-21T04:15:04.261265+0000","last_active":"2023-10-21T05:10:51.623001+0000","last_peered":"2023-10-21T05:10:51.623001+0000","last_clean":"2023-10-21T05:10:51.623001+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:10:51.623001+0000","last_undegraded":"2023-10-21T05:10:51.623001+0000","last_fullsized":"2023-10-21T05:10:51.623001+0000","mapping_epoch":15,"log_start":"78'443613","ondisk_log_start":"78'443613","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:39:42.118928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6876164096,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":453736,"num_read_kb":1785116,"num_write":451901,"num_write_kb":1807604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"79'15006","reported_seq":20747,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:49.828049+0000","last_change":"2023-10-21T04:05:01.187428+0000","last_active":"2023-10-21T05:10:49.828049+0000","last_peered":"2023-10-21T05:10:49.828049+0000","last_clean":"2023-10-21T05:10:49.828049+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:10:49.828049+0000","last_undegraded":"2023-10-21T05:10:49.828049+0000","last_fullsized":"2023-10-21T05:10:49.828049+0000","mapping_epoch":12,"log_start":"75'5000","ondisk_log_start":"75'5000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:02:28.019743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"79'449052","reported_seq":894362,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.912691+0000","last_change":"2023-10-21T04:15:04.363768+0000","last_active":"2023-10-21T05:10:51.912691+0000","last_peered":"2023-10-21T05:10:51.912691+0000","last_clean":"2023-10-21T05:10:51.912691+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:10:51.912691+0000","last_undegraded":"2023-10-21T05:10:51.912691+0000","last_fullsized":"2023-10-21T05:10:51.912691+0000","mapping_epoch":15,"log_start":"78'439016","ondisk_log_start":"78'439016","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:10:25.585633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6782939136,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":446887,"num_read_kb":1757444,"num_write":447297,"num_write_kb":1789188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'282","reported_seq":5986,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:49.845481+0000","last_change":"2023-10-21T04:05:01.189879+0000","last_active":"2023-10-21T05:10:49.845481+0000","last_peered":"2023-10-21T05:10:49.845481+0000","last_clean":"2023-10-21T05:10:49.845481+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:10:49.845481+0000","last_undegraded":"2023-10-21T05:10:49.845481+0000","last_fullsized":"2023-10-21T05:10:49.845481+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:41:43.561612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"79'466266","reported_seq":929320,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.639798+0000","last_change":"2023-10-21T04:15:04.256298+0000","last_active":"2023-10-21T05:10:51.639798+0000","last_peered":"2023-10-21T05:10:51.639798+0000","last_clean":"2023-10-21T05:10:51.639798+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:10:51.639798+0000","last_undegraded":"2023-10-21T05:10:51.639798+0000","last_fullsized":"2023-10-21T05:10:51.639798+0000","mapping_epoch":15,"log_start":"78'456223","ondisk_log_start":"78'456223","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:42:06.826330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7069523968,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":464744,"num_read_kb":1828392,"num_write":464402,"num_write_kb":1857608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'7","reported_seq":136,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-21T05:09:26.476861+0000","last_change":"2023-10-21T04:05:01.190036+0000","last_active":"2023-10-21T05:09:26.476861+0000","last_peered":"2023-10-21T05:09:26.476861+0000","last_clean":"2023-10-21T05:09:26.476861+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:09:26.476861+0000","last_undegraded":"2023-10-21T05:09:26.476861+0000","last_fullsized":"2023-10-21T05:09:26.476861+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:52:35.513076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"79'437805","reported_seq":873565,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.913192+0000","last_change":"2023-10-21T04:15:04.254514+0000","last_active":"2023-10-21T05:10:51.913192+0000","last_peered":"2023-10-21T05:10:51.913192+0000","last_clean":"2023-10-21T05:10:51.913192+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:10:51.913192+0000","last_undegraded":"2023-10-21T05:10:51.913192+0000","last_fullsized":"2023-10-21T05:10:51.913192+0000","mapping_epoch":15,"log_start":"78'427716","ondisk_log_start":"78'427716","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:55.129639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6629298195,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":437326,"num_read_kb":1719876,"num_write":436059,"num_write_kb":1744230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"79'3041","reported_seq":4298,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:49.827350+0000","last_change":"2023-10-21T04:05:01.186422+0000","last_active":"2023-10-21T05:10:49.827350+0000","last_peered":"2023-10-21T05:10:49.827350+0000","last_clean":"2023-10-21T05:10:49.827350+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:10:49.827350+0000","last_undegraded":"2023-10-21T05:10:49.827350+0000","last_fullsized":"2023-10-21T05:10:49.827350+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":3041,"log_dups_size":0,"ondisk_log_size":3041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:30:32.067265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":238,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-21T05:09:21.390470+0000","last_change":"2023-10-21T04:04:56.457261+0000","last_active":"2023-10-21T05:09:21.390470+0000","last_peered":"2023-10-21T05:09:21.390470+0000","last_clean":"2023-10-21T05:09:21.390470+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:09:21.390470+0000","last_undegraded":"2023-10-21T05:09:21.390470+0000","last_fullsized":"2023-10-21T05:09:21.390470+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:04:55.099596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:04:55.099596+0000","last_clean_scrub_stamp":"2023-10-21T04:04:55.099596+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:20:18.949598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27357925395,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1802693,"num_read_kb":7090828,"num_write":1799659,"num_write_kb":7198630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14743121920,"data_stored":16981790758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40259,"ondisk_log_size":40259,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":35413,"ondisk_log_size":35413,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739164,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4721108,"kb_used_data":3666568,"kb_used_omap":373,"kb_used_meta":1054154,"kb_avail":89650732,"statfs":{"total":96636764160,"available":91802349568,"internally_reserved":0,"allocated":3754565632,"data_stored":4326460737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1079454256},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,2,1,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.26200000000000001,"5min":0.39200000000000002,"15min":0.39400000000000002},"min":{"1min":0.17899999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.42199999999999999,"5min":4.8689999999999998,"15min":5.9009999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.40400000000000003,"15min":0.40300000000000002},"min":{"1min":0.19500000000000001,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":5.9349999999999996},"last":0.34799999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.66400000000000003,"15min":0.57299999999999995},"min":{"1min":0.20699999999999999,"5min":0.20100000000000001,"15min":0.17899999999999999},"max":{"1min":5.4100000000000001,"5min":14.318,"15min":14.318},"last":0.20399999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.70999999999999996,"15min":0.63200000000000001},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":5.4450000000000003,"5min":14.343,"15min":14.343},"last":0.25}]}]},{"osd":2,"up_from":8,"seq":34359739163,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9163068,"kb_used_data":7198772,"kb_used_omap":724,"kb_used_meta":1963563,"kb_avail":85208772,"statfs":{"total":96636764160,"available":87253782528,"internally_reserved":0,"allocated":7371542528,"data_stored":8490424105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":2010689138},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.59999999999999998,"15min":0.435},"min":{"1min":0.17100000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":3.0699999999999998,"5min":5.8899999999999997,"15min":6.383},"last":0.316},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.33600000000000002,"15min":0.45600000000000002},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.56499999999999995,"5min":2.8610000000000002,"15min":6.0289999999999999},"last":0.221}]},{"osd":1,"last update":"Sat Oct 21 05:10:46 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.35999999999999999,"15min":0.496},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":0.56699999999999995,"5min":1.7909999999999999,"15min":26.620999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.41499999999999998,"15min":0.627},"min":{"1min":0.245,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":0.50900000000000001,"5min":2.9929999999999999,"15min":26.596},"last":0.29299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739162,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4407892,"kb_used_data":3534668,"kb_used_omap":359,"kb_used_meta":872856,"kb_avail":89963948,"statfs":{"total":96636764160,"available":92123082752,"internally_reserved":0,"allocated":3619500032,"data_stored":4165730473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":893804578},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,10,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.38400000000000001,"15min":0.46999999999999997},"min":{"1min":0.186,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.51300000000000001,"5min":2.6320000000000001,"15min":13.757},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.0099999999999998,"5min":5.1459999999999999,"15min":13.183999999999999},"last":0.46000000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.55100000000000005},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.20599999999999999},"max":{"1min":1.782,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.36799999999999999,"15min":0.56399999999999995},"min":{"1min":0.23300000000000001,"5min":0.19800000000000001,"15min":0.193},"max":{"1min":1.462,"5min":2.6640000000000001,"15min":19.521999999999998},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3753181184,"data_stored":4325638144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3618226176,"data_stored":4165103635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7371714560,"data_stored":8491048979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:10:54.347 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T05:10:54.878 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:10:54.879 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":79,"fsid":"57c43eaf-d2a8-4c31-813f-7882b2f380d6","created":"2023-10-21T04:04:44.062479+0000","modified":"2023-10-21T05:10:30.298956+0000","last_up_change":"2023-10-21T04:04:54.047207+0000","last_in_change":"2023-10-21T04:04:44.928236+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T04:04:54.882624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-21T04:04:59.332315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-10-21T04:05:05.222624+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":71,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7bb86598-0c9a-42d0-89ca-b2506175e59d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6801","nonce":70176460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6803","nonce":70176460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6807","nonce":70176460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":70176460},{"type":"v1","addr":"172.21.15.119:6805","nonce":70176460}]},"public_addr":"172.21.15.119:6801/70176460","cluster_addr":"172.21.15.119:6803/70176460","heartbeat_back_addr":"172.21.15.119:6807/70176460","heartbeat_front_addr":"172.21.15.119:6805/70176460","state":["exists","up"]},{"osd":1,"uuid":"c8b63612-e8a0-4b1b-bf22-e4befb5787b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6817","nonce":2575111078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6819","nonce":2575111078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6823","nonce":2575111078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2575111078},{"type":"v1","addr":"172.21.15.119:6821","nonce":2575111078}]},"public_addr":"172.21.15.119:6817/2575111078","cluster_addr":"172.21.15.119:6819/2575111078","heartbeat_back_addr":"172.21.15.119:6823/2575111078","heartbeat_front_addr":"172.21.15.119:6821/2575111078","state":["exists","up"]},{"osd":2,"uuid":"bdf66e75-d9b7-40fe-bf4d-67a17adc0a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6809","nonce":1848266345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6811","nonce":1848266345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6815","nonce":1848266345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1848266345},{"type":"v1","addr":"172.21.15.119:6813","nonce":1848266345}]},"public_addr":"172.21.15.119:6809/1848266345","cluster_addr":"172.21.15.119:6811/1848266345","heartbeat_back_addr":"172.21.15.119:6815/1848266345","heartbeat_front_addr":"172.21.15.119:6813/1848266345","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:04:50.447383+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:04:50.187584+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:04:50.446482+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/161644020":"2023-10-21T05:22:15.338155+0000","172.21.15.119:0/1485658198":"2023-10-21T05:18:39.147632+0000","172.21.15.119:0/2943851828":"2023-10-21T05:11:37.958314+0000","172.21.15.119:0/2732595255":"2023-10-21T05:12:45.850393+0000","172.21.15.119:0/1214594455":"2023-10-21T05:16:11.134885+0000","172.21.15.119:0/1819191538":"2023-10-21T05:13:54.243082+0000","172.21.15.119:0/4093318189":"2023-10-21T05:15:02.985723+0000","172.21.15.119:0/843344947":"2023-10-21T05:20:55.678241+0000","172.21.15.119:0/1332372757":"2023-10-21T05:17:30.285042+0000","172.21.15.119:0/2726235268":"2023-10-21T05:19:47.384262+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T05:10:55.921 INFO:tasks.ceph:Scrubbing osd.0 2023-10-21T05:10:55.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-21T05:10:56.156 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2023-10-21T05:10:56.156 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-21T05:10:56.157 INFO:teuthology.orchestra.run.smithi119.stdout:} 2023-10-21T05:10:56.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-21T05:10:56.655 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:56.648+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:10:56.656 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:56.648+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:10:56.656 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:56.648+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:10:56.656 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:10:56.648+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:10:56.979 INFO:teuthology.orchestra.run.smithi119.stderr:instructed osd(s) 0 to deep-scrub 2023-10-21T05:10:57.022 INFO:tasks.ceph:Scrubbing osd.1 2023-10-21T05:10:57.023 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-21T05:10:57.251 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2023-10-21T05:10:57.251 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-21T05:10:57.251 INFO:teuthology.orchestra.run.smithi119.stdout:} 2023-10-21T05:10:57.285 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-21T05:10:57.788 INFO:teuthology.orchestra.run.smithi119.stderr:instructed osd(s) 1 to deep-scrub 2023-10-21T05:10:57.831 INFO:tasks.ceph:Scrubbing osd.2 2023-10-21T05:10:57.831 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-21T05:10:58.066 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2023-10-21T05:10:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-21T05:10:58.067 INFO:teuthology.orchestra.run.smithi119.stdout:} 2023-10-21T05:10:58.109 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-21T05:10:58.602 INFO:teuthology.orchestra.run.smithi119.stderr:instructed osd(s) 2 to deep-scrub 2023-10-21T05:10:58.644 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:10:59.171 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:10:59.172 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:10:59.212 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2057,"stamp":"2023-10-21T05:10:57.625887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27365940456,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1823037,"num_read_kb":7114993,"num_write":1802508,"num_write_kb":7209011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75799,"ondisk_log_size":75799,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18358496,"kb_used_data":14416456,"kb_used_omap":1457,"kb_used_meta":3940558,"kb_avail":264757024,"statfs":{"total":289910292480,"available":271111192576,"internally_reserved":0,"allocated":14762450944,"data_stored":16999458067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":4035131588},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,9,7,17,13,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23830528,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6586,"num_read_kb":26232,"num_write":6557,"num_write_kb":26228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001874"},"pg_stats":[{"pgid":"2.7","version":"79'8994","reported_seq":12403,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:56.660444+0000","last_change":"2023-10-21T04:05:01.187428+0000","last_active":"2023-10-21T05:10:56.660444+0000","last_peered":"2023-10-21T05:10:56.660444+0000","last_clean":"2023-10-21T05:10:56.660444+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:10:56.660444+0000","last_undegraded":"2023-10-21T05:10:56.660444+0000","last_fullsized":"2023-10-21T05:10:56.660444+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":8994,"log_dups_size":0,"ondisk_log_size":8994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:46:46.175450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"59'31","reported_seq":976,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-21T05:10:06.644676+0000","last_change":"2023-10-21T04:05:01.187482+0000","last_active":"2023-10-21T05:10:06.644676+0000","last_peered":"2023-10-21T05:10:06.644676+0000","last_clean":"2023-10-21T05:10:06.644676+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:10:06.644676+0000","last_undegraded":"2023-10-21T05:10:06.644676+0000","last_fullsized":"2023-10-21T05:10:06.644676+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:11:54.174741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'14854","reported_seq":20379,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:56.686088+0000","last_change":"2023-10-21T04:05:01.187628+0000","last_active":"2023-10-21T05:10:56.686088+0000","last_peered":"2023-10-21T05:10:56.686088+0000","last_clean":"2023-10-21T05:10:56.686088+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:10:56.686088+0000","last_undegraded":"2023-10-21T05:10:56.686088+0000","last_fullsized":"2023-10-21T05:10:56.686088+0000","mapping_epoch":12,"log_start":"75'4800","ondisk_log_start":"75'4800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:50:24.700355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"79'3071","reported_seq":4794,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:56.612066+0000","last_change":"2023-10-21T04:05:01.188633+0000","last_active":"2023-10-21T05:10:56.612066+0000","last_peered":"2023-10-21T05:10:56.612066+0000","last_clean":"2023-10-21T05:10:56.612066+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:10:56.612066+0000","last_undegraded":"2023-10-21T05:10:56.612066+0000","last_fullsized":"2023-10-21T05:10:56.612066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":3071,"log_dups_size":0,"ondisk_log_size":3071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:08:20.102163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"79'454251","reported_seq":906924,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:56.673177+0000","last_change":"2023-10-21T04:15:04.261265+0000","last_active":"2023-10-21T05:10:56.673177+0000","last_peered":"2023-10-21T05:10:56.673177+0000","last_clean":"2023-10-21T05:10:56.673177+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:10:56.673177+0000","last_undegraded":"2023-10-21T05:10:56.673177+0000","last_fullsized":"2023-10-21T05:10:56.673177+0000","mapping_epoch":15,"log_start":"78'444213","ondisk_log_start":"78'444213","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:39:42.118928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6878146560,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":454303,"num_read_kb":1787384,"num_write":452448,"num_write_kb":1809792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"79'15011","reported_seq":20757,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:54.947869+0000","last_change":"2023-10-21T04:05:01.187428+0000","last_active":"2023-10-21T05:10:54.947869+0000","last_peered":"2023-10-21T05:10:54.947869+0000","last_clean":"2023-10-21T05:10:54.947869+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:10:54.947869+0000","last_undegraded":"2023-10-21T05:10:54.947869+0000","last_fullsized":"2023-10-21T05:10:54.947869+0000","mapping_epoch":12,"log_start":"75'5000","ondisk_log_start":"75'5000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:02:28.019743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"79'449570","reported_seq":895402,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:56.908398+0000","last_change":"2023-10-21T04:15:04.363768+0000","last_active":"2023-10-21T05:10:56.908398+0000","last_peered":"2023-10-21T05:10:56.908398+0000","last_clean":"2023-10-21T05:10:56.908398+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:10:56.908398+0000","last_undegraded":"2023-10-21T05:10:56.908398+0000","last_fullsized":"2023-10-21T05:10:56.908398+0000","mapping_epoch":15,"log_start":"78'439516","ondisk_log_start":"78'439516","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:10:25.585633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6784798720,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":447406,"num_read_kb":1759512,"num_write":447815,"num_write_kb":1791260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"75'282","reported_seq":5991,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:55.040101+0000","last_change":"2023-10-21T04:05:01.189879+0000","last_active":"2023-10-21T05:10:55.040101+0000","last_peered":"2023-10-21T05:10:55.040101+0000","last_clean":"2023-10-21T05:10:55.040101+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:10:55.040101+0000","last_undegraded":"2023-10-21T05:10:55.040101+0000","last_fullsized":"2023-10-21T05:10:55.040101+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:41:43.561612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"79'466798","reported_seq":930401,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:56.668383+0000","last_change":"2023-10-21T04:15:04.256298+0000","last_active":"2023-10-21T05:10:56.668383+0000","last_peered":"2023-10-21T05:10:56.668383+0000","last_clean":"2023-10-21T05:10:56.668383+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:10:56.668383+0000","last_undegraded":"2023-10-21T05:10:56.668383+0000","last_fullsized":"2023-10-21T05:10:56.668383+0000","mapping_epoch":15,"log_start":"78'456723","ondisk_log_start":"78'456723","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:42:06.826330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7071506432,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":465291,"num_read_kb":1830568,"num_write":464934,"num_write_kb":1859736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'7","reported_seq":136,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-21T05:09:26.476861+0000","last_change":"2023-10-21T04:05:01.190036+0000","last_active":"2023-10-21T05:09:26.476861+0000","last_peered":"2023-10-21T05:09:26.476861+0000","last_clean":"2023-10-21T05:09:26.476861+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:09:26.476861+0000","last_undegraded":"2023-10-21T05:09:26.476861+0000","last_fullsized":"2023-10-21T05:09:26.476861+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:52:35.513076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"79'438251","reported_seq":874557,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:56.916893+0000","last_change":"2023-10-21T04:15:04.254514+0000","last_active":"2023-10-21T05:10:56.916893+0000","last_peered":"2023-10-21T05:10:56.916893+0000","last_clean":"2023-10-21T05:10:56.916893+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:10:56.916893+0000","last_undegraded":"2023-10-21T05:10:56.916893+0000","last_fullsized":"2023-10-21T05:10:56.916893+0000","mapping_epoch":15,"log_start":"78'428216","ondisk_log_start":"78'428216","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:55.129639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6630895635,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":437872,"num_read_kb":1722052,"num_write":436505,"num_write_kb":1746014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"79'3050","reported_seq":4308,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:10:56.637319+0000","last_change":"2023-10-21T04:05:01.186422+0000","last_active":"2023-10-21T05:10:56.637319+0000","last_peered":"2023-10-21T05:10:56.637319+0000","last_clean":"2023-10-21T05:10:56.637319+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:10:56.637319+0000","last_undegraded":"2023-10-21T05:10:56.637319+0000","last_fullsized":"2023-10-21T05:10:56.637319+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:00.160431+0000","last_clean_scrub_stamp":"2023-10-21T04:05:00.160431+0000","objects_scrubbed":0,"log_size":3050,"log_dups_size":0,"ondisk_log_size":3050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:30:32.067265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":238,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-10-21T05:09:21.390470+0000","last_change":"2023-10-21T04:04:56.457261+0000","last_active":"2023-10-21T05:09:21.390470+0000","last_peered":"2023-10-21T05:09:21.390470+0000","last_clean":"2023-10-21T05:09:21.390470+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:09:21.390470+0000","last_undegraded":"2023-10-21T05:09:21.390470+0000","last_fullsized":"2023-10-21T05:09:21.390470+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:04:55.099596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:04:55.099596+0000","last_clean_scrub_stamp":"2023-10-21T04:04:55.099596+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:20:18.949598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27365347347,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1804872,"num_read_kb":7099516,"num_write":1801702,"num_write_kb":7206802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14759919616,"data_stored":16998588454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40202,"ondisk_log_size":40202,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":35500,"ondisk_log_size":35500,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739165,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4747624,"kb_used_data":3670880,"kb_used_omap":373,"kb_used_meta":1076362,"kb_avail":89624216,"statfs":{"total":96636764160,"available":91775197184,"internally_reserved":0,"allocated":3758981120,"data_stored":4330876225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1102195248},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,4,10,8,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.38300000000000001,"15min":0.41099999999999998},"min":{"1min":0.189,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.191,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":4.6509999999999998,"5min":5.0060000000000002,"15min":5.9349999999999996},"last":0.23300000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:10:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.69699999999999995,"15min":0.57699999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":2.488,"5min":14.318,"15min":14.318},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.74099999999999999,"15min":0.63900000000000001},"min":{"1min":0.25,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.5209999999999999,"5min":14.343,"15min":14.343},"last":0.29999999999999999}]}]},{"osd":2,"up_from":8,"seq":34359739164,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9176860,"kb_used_data":7206996,"kb_used_omap":724,"kb_used_meta":1969131,"kb_avail":85194980,"statfs":{"total":96636764160,"available":87239659520,"internally_reserved":0,"allocated":7379963904,"data_stored":8498845481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":2016390770},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.59999999999999998,"15min":0.435},"min":{"1min":0.17100000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":3.0699999999999998,"5min":5.8899999999999997,"15min":6.383},"last":0.316},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.33600000000000002,"15min":0.45600000000000002},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.56499999999999995,"5min":2.8610000000000002,"15min":6.0289999999999999},"last":0.221}]},{"osd":1,"last update":"Sat Oct 21 05:10:46 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.35999999999999999,"15min":0.496},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":0.56699999999999995,"5min":1.7909999999999999,"15min":26.620999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.41499999999999998,"15min":0.627},"min":{"1min":0.245,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":0.50900000000000001,"5min":2.9929999999999999,"15min":26.596},"last":0.29299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739163,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4434012,"kb_used_data":3538580,"kb_used_omap":359,"kb_used_meta":895064,"kb_avail":89937828,"statfs":{"total":96636764160,"available":92096335872,"internally_reserved":0,"allocated":3623505920,"data_stored":4169736361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":916545570},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,0,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.38400000000000001,"15min":0.46999999999999997},"min":{"1min":0.186,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.51300000000000001,"5min":2.6320000000000001,"15min":13.757},"last":0.443},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.0099999999999998,"5min":5.1459999999999999,"15min":13.183999999999999},"last":0.215}]},{"osd":2,"last update":"Sat Oct 21 05:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.55100000000000005},"min":{"1min":0.23100000000000001,"5min":0.21299999999999999,"15min":0.20599999999999999},"max":{"1min":1.782,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.36799999999999999,"15min":0.56399999999999995},"min":{"1min":0.23300000000000001,"5min":0.19800000000000001,"15min":0.193},"max":{"1min":1.462,"5min":2.6640000000000001,"15min":19.521999999999998},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3757596672,"data_stored":4330053632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3622174720,"data_stored":4169052179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7380148224,"data_stored":8499482643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:10:59.215 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-21T04:05:00.160431+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=0, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.215 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-21T04:05:00.160431+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=0, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.216 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-21T04:05:00.160431+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=0, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.216 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-21T04:05:00.160431+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=0, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.216 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.217 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-21T04:05:00.160431+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=0, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.217 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.217 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-21T04:05:00.160431+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=0, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.218 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.218 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-21T04:05:00.160431+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=0, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.219 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.219 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-21T04:05:00.160431+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=0, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.219 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-21T04:04:55.099596+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=4, tm_sec=55, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:59.220 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:11:17.403 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:11:17.396+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:11:17.403 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:11:17.400+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:11:19.220 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:11:19.689 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:11:19.725 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2068,"stamp":"2023-10-21T05:11:19.629700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27394415848,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1830823,"num_read_kb":7146009,"num_write":1810286,"num_write_kb":7240123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75962,"ondisk_log_size":75962,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18560348,"kb_used_data":14478212,"kb_used_omap":1457,"kb_used_meta":4080654,"kb_avail":264555172,"statfs":{"total":289910292480,"available":270904496128,"internally_reserved":0,"allocated":14825689088,"data_stored":17062696211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":4178589892},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,7,8,7,5,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13520896,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3718,"num_read_kb":14816,"num_write":3732,"num_write_kb":14928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002168"},"pg_stats":[{"pgid":"2.7","version":"79'9045","reported_seq":12475,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:16.641384+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:11:16.641384+0000","last_peered":"2023-10-21T05:11:16.641384+0000","last_clean":"2023-10-21T05:11:16.641384+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:11:16.641384+0000","last_undegraded":"2023-10-21T05:11:16.641384+0000","last_fullsized":"2023-10-21T05:11:16.641384+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":994,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:08.750140+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:11:08.750140+0000","last_peered":"2023-10-21T05:11:08.750140+0000","last_clean":"2023-10-21T05:11:08.750140+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:11:08.750140+0000","last_undegraded":"2023-10-21T05:11:08.750140+0000","last_fullsized":"2023-10-21T05:11:08.750140+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'14929","reported_seq":20483,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:16.664276+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:11:16.664276+0000","last_peered":"2023-10-21T05:11:16.664276+0000","last_clean":"2023-10-21T05:11:16.664276+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:11:16.664276+0000","last_undegraded":"2023-10-21T05:11:16.664276+0000","last_fullsized":"2023-10-21T05:11:16.664276+0000","mapping_epoch":12,"log_start":"75'4900","ondisk_log_start":"75'4900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"79'3094","reported_seq":4830,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:16.572888+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:11:16.572888+0000","last_peered":"2023-10-21T05:11:16.572888+0000","last_clean":"2023-10-21T05:11:16.572888+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:11:16.572888+0000","last_undegraded":"2023-10-21T05:11:16.572888+0000","last_fullsized":"2023-10-21T05:11:16.572888+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3094,"log_dups_size":0,"ondisk_log_size":3094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"79'456241","reported_seq":910896,"reported_epoch":79,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:16.940814+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:11:16.940814+0000","last_peered":"2023-10-21T05:11:16.940814+0000","last_clean":"2023-10-21T05:11:16.940814+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:11:16.940814+0000","last_undegraded":"2023-10-21T05:11:16.940814+0000","last_fullsized":"2023-10-21T05:11:16.940814+0000","mapping_epoch":15,"log_start":"78'446213","ondisk_log_start":"78'446213","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":63,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 10s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6885191680,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":456269,"num_read_kb":1795220,"num_write":454400,"num_write_kb":1817600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"79'15086","reported_seq":20862,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:16.840045+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:11:16.840045+0000","last_peered":"2023-10-21T05:11:16.840045+0000","last_clean":"2023-10-21T05:11:16.840045+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:11:16.840045+0000","last_undegraded":"2023-10-21T05:11:16.840045+0000","last_fullsized":"2023-10-21T05:11:16.840045+0000","mapping_epoch":12,"log_start":"75'5000","ondisk_log_start":"75'5000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"79'451462","reported_seq":899217,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:17.007342+0000","last_change":"2023-10-21T05:11:14.695082+0000","last_active":"2023-10-21T05:11:17.007342+0000","last_peered":"2023-10-21T05:11:17.007342+0000","last_clean":"2023-10-21T05:11:17.007342+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:11:17.007342+0000","last_undegraded":"2023-10-21T05:11:17.007342+0000","last_fullsized":"2023-10-21T05:11:17.007342+0000","mapping_epoch":15,"log_start":"78'441416","ondisk_log_start":"78'441416","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6791725056,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":449325,"num_read_kb":1767172,"num_write":449707,"num_write_kb":1798828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6031,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:15.432559+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:11:15.432559+0000","last_peered":"2023-10-21T05:11:15.432559+0000","last_clean":"2023-10-21T05:11:15.432559+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:11:15.432559+0000","last_undegraded":"2023-10-21T05:11:15.432559+0000","last_fullsized":"2023-10-21T05:11:15.432559+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"79'468885","reported_seq":934521,"reported_epoch":79,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:16.922732+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:11:16.922732+0000","last_peered":"2023-10-21T05:11:16.922732+0000","last_clean":"2023-10-21T05:11:16.922732+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:11:16.922732+0000","last_undegraded":"2023-10-21T05:11:16.922732+0000","last_fullsized":"2023-10-21T05:11:16.922732+0000","mapping_epoch":15,"log_start":"78'458823","ondisk_log_start":"78'458823","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":88,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 11s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7078961152,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":467313,"num_read_kb":1838612,"num_write":466966,"num_write_kb":1867864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":152,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:03.630056+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:11:03.630056+0000","last_peered":"2023-10-21T05:11:03.630056+0000","last_clean":"2023-10-21T05:11:03.630056+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:11:03.630056+0000","last_undegraded":"2023-10-21T05:11:03.630056+0000","last_fullsized":"2023-10-21T05:11:03.630056+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"79'440171","reported_seq":878362,"reported_epoch":79,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:17.006395+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:11:17.006395+0000","last_peered":"2023-10-21T05:11:17.006395+0000","last_clean":"2023-10-21T05:11:17.006395+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:11:17.006395+0000","last_undegraded":"2023-10-21T05:11:17.006395+0000","last_fullsized":"2023-10-21T05:11:17.006395+0000","mapping_epoch":15,"log_start":"78'430116","ondisk_log_start":"78'430116","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":27,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 7s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6637944851,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":439751,"num_read_kb":1729528,"num_write":438407,"num_write_kb":1753622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"79'3071","reported_seq":4342,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:16.602365+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:11:16.602365+0000","last_peered":"2023-10-21T05:11:16.602365+0000","last_clean":"2023-10-21T05:11:16.602365+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:11:16.602365+0000","last_undegraded":"2023-10-21T05:11:16.602365+0000","last_fullsized":"2023-10-21T05:11:16.602365+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3071,"log_dups_size":0,"ondisk_log_size":3071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":249,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:01.852956+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:11:01.852956+0000","last_peered":"2023-10-21T05:11:01.852956+0000","last_clean":"2023-10-21T05:11:01.852956+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:11:01.852956+0000","last_undegraded":"2023-10-21T05:11:01.852956+0000","last_fullsized":"2023-10-21T05:11:01.852956+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27393822739,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1812658,"num_read_kb":7130532,"num_write":1809480,"num_write_kb":7237914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14823481344,"data_stored":17062150182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40191,"ondisk_log_size":40191,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":35672,"ondisk_log_size":35672,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739169,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4799440,"kb_used_data":3686600,"kb_used_omap":373,"kb_used_meta":1112458,"kb_avail":89572400,"statfs":{"total":96636764160,"available":91722137600,"internally_reserved":0,"allocated":3775078400,"data_stored":4346973505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1139157552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,3,3,4,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.38300000000000001,"15min":0.41099999999999998},"min":{"1min":0.189,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.193},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.191,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":4.6509999999999998,"5min":5.0060000000000002,"15min":5.9349999999999996},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:10:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.69699999999999995,"15min":0.57699999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":2.488,"5min":14.318,"15min":14.318},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.74099999999999999,"15min":0.63900000000000001},"min":{"1min":0.25,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.5209999999999999,"5min":14.343,"15min":14.343},"last":0.40100000000000002}]}]},{"osd":2,"up_from":8,"seq":34359739168,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9275728,"kb_used_data":7237896,"kb_used_omap":724,"kb_used_meta":2037099,"kb_avail":85096112,"statfs":{"total":96636764160,"available":87138418688,"internally_reserved":0,"allocated":7411605504,"data_stored":8530487081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":2085990002},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.59999999999999998,"15min":0.435},"min":{"1min":0.17100000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":3.0699999999999998,"5min":5.8899999999999997,"15min":6.383},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.33600000000000002,"15min":0.45600000000000002},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.56499999999999995,"5min":2.8610000000000002,"15min":6.0289999999999999},"last":0.30499999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:10:46 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.35999999999999999,"15min":0.496},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":0.56699999999999995,"5min":1.7909999999999999,"15min":26.620999999999999},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.41499999999999998,"15min":0.627},"min":{"1min":0.245,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":0.50900000000000001,"5min":2.9929999999999999,"15min":26.596},"last":0.34100000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739167,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4485180,"kb_used_data":3553716,"kb_used_omap":359,"kb_used_meta":931096,"kb_avail":89886660,"statfs":{"total":96636764160,"available":92043939840,"internally_reserved":0,"allocated":3639005184,"data_stored":4185235625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":953442338},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.376,"15min":0.46800000000000003},"min":{"1min":0.16800000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.52600000000000002,"5min":2.6320000000000001,"15min":13.757},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.21099999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9550000000000001,"5min":5.1459999999999999,"15min":13.183999999999999},"last":0.33500000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.501,"15min":0.55500000000000005},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.409,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.224},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.377,"15min":0.55100000000000005},"min":{"1min":0.23799999999999999,"5min":0.19800000000000001,"15min":0.193},"max":{"1min":1.3009999999999999,"5min":2.6640000000000001,"15min":19.521999999999998},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3773923328,"data_stored":4346380288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3637710848,"data_stored":4184588307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7411847168,"data_stored":8531181587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:19.727 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:19.727 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:19.728 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:19.728 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:19.728 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:11:39.729 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:40.202 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:11:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:11:40.229 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2078,"stamp":"2023-10-21T05:11:39.632981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27420912872,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1838125,"num_read_kb":7175065,"num_write":1817467,"num_write_kb":7268847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75951,"ondisk_log_size":75951,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18789700,"kb_used_data":14535148,"kb_used_omap":1457,"kb_used_meta":4253070,"kb_avail":264325820,"statfs":{"total":289910292480,"available":270669639680,"internally_reserved":0,"allocated":14883991552,"data_stored":17120998675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":4355143876},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,6,5,12,17,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13246464,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3568,"num_read_kb":14176,"num_write":3628,"num_write_kb":14512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002004"},"pg_stats":[{"pgid":"2.7","version":"79'9085","reported_seq":12527,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:35.908169+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:11:35.908169+0000","last_peered":"2023-10-21T05:11:35.908169+0000","last_clean":"2023-10-21T05:11:35.908169+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:11:35.908169+0000","last_undegraded":"2023-10-21T05:11:35.908169+0000","last_fullsized":"2023-10-21T05:11:35.908169+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9085,"log_dups_size":0,"ondisk_log_size":9085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":994,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:08.750140+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:11:08.750140+0000","last_peered":"2023-10-21T05:11:08.750140+0000","last_clean":"2023-10-21T05:11:08.750140+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:11:08.750140+0000","last_undegraded":"2023-10-21T05:11:08.750140+0000","last_fullsized":"2023-10-21T05:11:08.750140+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'14995","reported_seq":20569,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:35.908489+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:11:35.908489+0000","last_peered":"2023-10-21T05:11:35.908489+0000","last_clean":"2023-10-21T05:11:35.908489+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:11:35.908489+0000","last_undegraded":"2023-10-21T05:11:35.908489+0000","last_fullsized":"2023-10-21T05:11:35.908489+0000","mapping_epoch":12,"log_start":"75'4900","ondisk_log_start":"75'4900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"79'3107","reported_seq":4847,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:35.412706+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:11:35.412706+0000","last_peered":"2023-10-21T05:11:35.412706+0000","last_clean":"2023-10-21T05:11:35.412706+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:11:35.412706+0000","last_undegraded":"2023-10-21T05:11:35.412706+0000","last_fullsized":"2023-10-21T05:11:35.412706+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"79'458220","reported_seq":914771,"reported_epoch":79,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:37.020132+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:11:37.020132+0000","last_peered":"2023-10-21T05:11:37.020132+0000","last_clean":"2023-10-21T05:11:37.020132+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:11:37.020132+0000","last_undegraded":"2023-10-21T05:11:37.020132+0000","last_fullsized":"2023-10-21T05:11:37.020132+0000","mapping_epoch":15,"log_start":"78'448214","ondisk_log_start":"78'448214","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":238,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 30s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6892150784,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":458150,"num_read_kb":1802712,"num_write":456265,"num_write_kb":1825060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"79'15145","reported_seq":20941,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:35.908102+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:11:35.908102+0000","last_peered":"2023-10-21T05:11:35.908102+0000","last_clean":"2023-10-21T05:11:35.908102+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:11:35.908102+0000","last_undegraded":"2023-10-21T05:11:35.908102+0000","last_fullsized":"2023-10-21T05:11:35.908102+0000","mapping_epoch":12,"log_start":"75'5100","ondisk_log_start":"75'5100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"79'453218","reported_seq":902802,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:37.062265+0000","last_change":"2023-10-21T05:11:27.664397+0000","last_active":"2023-10-21T05:11:37.062265+0000","last_peered":"2023-10-21T05:11:37.062265+0000","last_clean":"2023-10-21T05:11:37.062265+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:11:37.062265+0000","last_undegraded":"2023-10-21T05:11:37.062265+0000","last_fullsized":"2023-10-21T05:11:37.062265+0000","mapping_epoch":15,"log_start":"78'443216","ondisk_log_start":"78'443216","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6798147584,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":451153,"num_read_kb":1774412,"num_write":451463,"num_write_kb":1805852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6051,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:35.910236+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:11:35.910236+0000","last_peered":"2023-10-21T05:11:35.910236+0000","last_clean":"2023-10-21T05:11:35.910236+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:11:35.910236+0000","last_undegraded":"2023-10-21T05:11:35.910236+0000","last_fullsized":"2023-10-21T05:11:35.910236+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"79'470838","reported_seq":938374,"reported_epoch":79,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:37.002989+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:11:37.002989+0000","last_peered":"2023-10-21T05:11:37.002989+0000","last_clean":"2023-10-21T05:11:37.002989+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:11:37.002989+0000","last_undegraded":"2023-10-21T05:11:37.002989+0000","last_fullsized":"2023-10-21T05:11:37.002989+0000","mapping_epoch":15,"log_start":"78'460823","ondisk_log_start":"78'460823","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":233,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 31s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7085867008,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":469196,"num_read_kb":1846108,"num_write":468829,"num_write_kb":1875316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":153,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:03.630166+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:11:03.630166+0000","last_peered":"2023-10-21T05:11:03.630166+0000","last_clean":"2023-10-21T05:11:03.630166+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:11:03.630166+0000","last_undegraded":"2023-10-21T05:11:03.630166+0000","last_fullsized":"2023-10-21T05:11:03.630166+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"79'441981","reported_seq":881897,"reported_epoch":79,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:37.060212+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:11:37.060212+0000","last_peered":"2023-10-21T05:11:37.060212+0000","last_clean":"2023-10-21T05:11:37.060212+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:11:37.060212+0000","last_undegraded":"2023-10-21T05:11:37.060212+0000","last_fullsized":"2023-10-21T05:11:37.060212+0000","mapping_epoch":15,"log_start":"78'431916","ondisk_log_start":"78'431916","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":205,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 27s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6644154387,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":441461,"num_read_kb":1736356,"num_write":440104,"num_write_kb":1760410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"79'3085","reported_seq":4360,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:35.416669+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:11:35.416669+0000","last_peered":"2023-10-21T05:11:35.416669+0000","last_clean":"2023-10-21T05:11:35.416669+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:11:35.416669+0000","last_undegraded":"2023-10-21T05:11:35.416669+0000","last_fullsized":"2023-10-21T05:11:35.416669+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":250,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:01.853225+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:11:01.853225+0000","last_peered":"2023-10-21T05:11:01.853225+0000","last_clean":"2023-10-21T05:11:01.853225+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:11:01.853225+0000","last_undegraded":"2023-10-21T05:11:01.853225+0000","last_fullsized":"2023-10-21T05:11:01.853225+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27420319763,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1819960,"num_read_kb":7159588,"num_write":1816661,"num_write_kb":7266638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14882287616,"data_stored":17120956454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40088,"ondisk_log_size":40088,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":35764,"ondisk_log_size":35764,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739173,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4850428,"kb_used_data":3701492,"kb_used_omap":373,"kb_used_meta":1148554,"kb_avail":89521412,"statfs":{"total":96636764160,"available":91669925888,"internally_reserved":0,"allocated":3790327808,"data_stored":4362222913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1176119856},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,6,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.38300000000000001,"15min":0.41099999999999998},"min":{"1min":0.189,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.317},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.191,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":4.6509999999999998,"5min":5.0060000000000002,"15min":5.9349999999999996},"last":0.27800000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:10:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.69699999999999995,"15min":0.57699999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":2.488,"5min":14.318,"15min":14.318},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.74099999999999999,"15min":0.63900000000000001},"min":{"1min":0.25,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.5209999999999999,"5min":14.343,"15min":14.343},"last":0.61899999999999999}]}]},{"osd":2,"up_from":8,"seq":34359739172,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9380164,"kb_used_data":7266236,"kb_used_omap":724,"kb_used_meta":2113195,"kb_avail":84991676,"statfs":{"total":96636764160,"available":87031476224,"internally_reserved":0,"allocated":7440625664,"data_stored":8559507241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":2163912306},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,1,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.59999999999999998,"15min":0.435},"min":{"1min":0.17100000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":3.0699999999999998,"5min":5.8899999999999997,"15min":6.383},"last":0.224},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.33600000000000002,"15min":0.45600000000000002},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.56499999999999995,"5min":2.8610000000000002,"15min":6.0289999999999999},"last":0.26100000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:10:46 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.35999999999999999,"15min":0.496},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":0.56699999999999995,"5min":1.7909999999999999,"15min":26.620999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.41499999999999998,"15min":0.627},"min":{"1min":0.245,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":0.50900000000000001,"5min":2.9929999999999999,"15min":26.596},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359739171,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4559108,"kb_used_data":3567420,"kb_used_omap":359,"kb_used_meta":991320,"kb_avail":89812732,"statfs":{"total":96636764160,"available":91968237568,"internally_reserved":0,"allocated":3653038080,"data_stored":4199268521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1015111714},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,2,3,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.376,"15min":0.46800000000000003},"min":{"1min":0.16800000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.52600000000000002,"5min":2.6320000000000001,"15min":13.757},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.21099999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9550000000000001,"5min":5.1459999999999999,"15min":13.183999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.501,"15min":0.55500000000000005},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.409,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.313},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.377,"15min":0.55100000000000005},"min":{"1min":0.23799999999999999,"5min":0.19800000000000001,"15min":0.193},"max":{"1min":1.3009999999999999,"5min":2.6640000000000001,"15min":19.521999999999998},"last":0.39300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3789185024,"data_stored":4361641984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3651862528,"data_stored":4198739987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7441240064,"data_stored":8560574483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:40.230 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:40.231 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:40.231 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:40.232 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:40.232 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:11:42.142 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:11:42.135+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:11:42.143 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:11:42.135+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:11:42.143 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:11:42.135+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:11:47.122 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:11:47.115+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:11:47.127 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:11:47.119+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:11:47.127 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:11:47.119+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:11:47.128 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:11:47.119+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:12:00.234 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:12:00.706 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:12:00.706 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:12:00.739 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2089,"stamp":"2023-10-21T05:11:59.636158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27448306920,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1845573,"num_read_kb":7204725,"num_write":1824891,"num_write_kb":7298543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75905,"ondisk_log_size":75905,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18978664,"kb_used_data":14595920,"kb_used_omap":1457,"kb_used_meta":4381262,"kb_avail":264136856,"statfs":{"total":289910292480,"available":270476140544,"internally_reserved":0,"allocated":14946222080,"data_stored":17183216387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":4486412484},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,7,9,18,15,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13398016,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3561,"num_read_kb":14140,"num_write":3641,"num_write_kb":14564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001945"},"pg_stats":[{"pgid":"2.7","version":"80'9105","reported_seq":12560,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:56.387790+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:11:56.387790+0000","last_peered":"2023-10-21T05:11:56.387790+0000","last_clean":"2023-10-21T05:11:56.387790+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:11:56.387790+0000","last_undegraded":"2023-10-21T05:11:56.387790+0000","last_fullsized":"2023-10-21T05:11:56.387790+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9105,"log_dups_size":0,"ondisk_log_size":9105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":995,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:08.750236+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:11:08.750236+0000","last_peered":"2023-10-21T05:11:08.750236+0000","last_clean":"2023-10-21T05:11:08.750236+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:11:08.750236+0000","last_undegraded":"2023-10-21T05:11:08.750236+0000","last_fullsized":"2023-10-21T05:11:08.750236+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"80'15027","reported_seq":20622,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:56.388728+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:11:56.388728+0000","last_peered":"2023-10-21T05:11:56.388728+0000","last_clean":"2023-10-21T05:11:56.388728+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:11:56.388728+0000","last_undegraded":"2023-10-21T05:11:56.388728+0000","last_fullsized":"2023-10-21T05:11:56.388728+0000","mapping_epoch":12,"log_start":"75'5000","ondisk_log_start":"75'5000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"80'3115","reported_seq":4860,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:53.403261+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:11:53.403261+0000","last_peered":"2023-10-21T05:11:53.403261+0000","last_clean":"2023-10-21T05:11:53.403261+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:11:53.403261+0000","last_undegraded":"2023-10-21T05:11:53.403261+0000","last_fullsized":"2023-10-21T05:11:53.403261+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3115,"log_dups_size":0,"ondisk_log_size":3115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"80'460143","reported_seq":918570,"reported_epoch":80,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:57.110118+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:11:57.110118+0000","last_peered":"2023-10-21T05:11:57.110118+0000","last_clean":"2023-10-21T05:11:57.110118+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:11:57.110118+0000","last_undegraded":"2023-10-21T05:11:57.110118+0000","last_fullsized":"2023-10-21T05:11:57.110118+0000","mapping_epoch":15,"log_start":"78'450114","ondisk_log_start":"78'450114","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":409,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 50s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6898876416,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":460012,"num_read_kb":1810112,"num_write":458084,"num_write_kb":1832336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"80'15220","reported_seq":21037,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:56.388064+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:11:56.388064+0000","last_peered":"2023-10-21T05:11:56.388064+0000","last_clean":"2023-10-21T05:11:56.388064+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:11:56.388064+0000","last_undegraded":"2023-10-21T05:11:56.388064+0000","last_fullsized":"2023-10-21T05:11:56.388064+0000","mapping_epoch":12,"log_start":"75'5200","ondisk_log_start":"75'5200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"80'455073","reported_seq":906591,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:57.149592+0000","last_change":"2023-10-21T05:11:54.791521+0000","last_active":"2023-10-21T05:11:57.149592+0000","last_peered":"2023-10-21T05:11:57.149592+0000","last_clean":"2023-10-21T05:11:57.149592+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:11:57.149592+0000","last_undegraded":"2023-10-21T05:11:57.149592+0000","last_fullsized":"2023-10-21T05:11:57.149592+0000","mapping_epoch":15,"log_start":"78'445016","ondisk_log_start":"78'445016","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6804930560,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":453080,"num_read_kb":1782092,"num_write":453318,"num_write_kb":1813272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6073,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:56.389415+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:11:56.389415+0000","last_peered":"2023-10-21T05:11:56.389415+0000","last_clean":"2023-10-21T05:11:56.389415+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:11:56.389415+0000","last_undegraded":"2023-10-21T05:11:56.389415+0000","last_fullsized":"2023-10-21T05:11:56.389415+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"80'472834","reported_seq":942303,"reported_epoch":80,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:57.109995+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:11:57.109995+0000","last_peered":"2023-10-21T05:11:57.109995+0000","last_clean":"2023-10-21T05:11:57.109995+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:11:57.109995+0000","last_undegraded":"2023-10-21T05:11:57.109995+0000","last_fullsized":"2023-10-21T05:11:57.109995+0000","mapping_epoch":15,"log_start":"78'462823","ondisk_log_start":"78'462823","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":363,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 51s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7092985856,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":471115,"num_read_kb":1853752,"num_write":470743,"num_write_kb":1882972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":154,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:27.148690+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:11:27.148690+0000","last_peered":"2023-10-21T05:11:27.148690+0000","last_clean":"2023-10-21T05:11:27.148690+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:11:27.148690+0000","last_undegraded":"2023-10-21T05:11:27.148690+0000","last_fullsized":"2023-10-21T05:11:27.148690+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"80'443918","reported_seq":885594,"reported_epoch":80,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:57.154457+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:11:57.154457+0000","last_peered":"2023-10-21T05:11:57.154457+0000","last_clean":"2023-10-21T05:11:57.154457+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:11:57.154457+0000","last_undegraded":"2023-10-21T05:11:57.154457+0000","last_fullsized":"2023-10-21T05:11:57.154457+0000","mapping_epoch":15,"log_start":"78'433916","ondisk_log_start":"78'433916","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":365,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 47s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6650920979,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":443201,"num_read_kb":1743292,"num_write":441940,"num_write_kb":1767754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"80'3093","reported_seq":4373,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:53.418741+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:11:53.418741+0000","last_peered":"2023-10-21T05:11:53.418741+0000","last_clean":"2023-10-21T05:11:53.418741+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:11:53.418741+0000","last_undegraded":"2023-10-21T05:11:53.418741+0000","last_fullsized":"2023-10-21T05:11:53.418741+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3093,"log_dups_size":0,"ondisk_log_size":3093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":251,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:22.038918+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:11:22.038918+0000","last_peered":"2023-10-21T05:11:22.038918+0000","last_clean":"2023-10-21T05:11:22.038918+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:11:22.038918+0000","last_undegraded":"2023-10-21T05:11:22.038918+0000","last_fullsized":"2023-10-21T05:11:22.038918+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27447713811,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1827408,"num_read_kb":7189248,"num_write":1824085,"num_write_kb":7296334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14943121408,"data_stored":17181790246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40099,"ondisk_log_size":40099,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":35707,"ondisk_log_size":35707,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739177,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4889544,"kb_used_data":3716480,"kb_used_omap":373,"kb_used_meta":1172682,"kb_avail":89482296,"statfs":{"total":96636764160,"available":91629871104,"internally_reserved":0,"allocated":3805675520,"data_stored":4377566353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1200826928},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,7,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.38300000000000001,"15min":0.41099999999999998},"min":{"1min":0.189,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.245},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.191,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":4.6509999999999998,"5min":5.0060000000000002,"15min":5.9349999999999996},"last":0.20499999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:10:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.69699999999999995,"15min":0.57699999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":2.488,"5min":14.318,"15min":14.318},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.74099999999999999,"15min":0.63900000000000001},"min":{"1min":0.25,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.5209999999999999,"5min":14.343,"15min":14.343},"last":0.51000000000000001}]}]},{"osd":2,"up_from":8,"seq":34359739176,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9497132,"kb_used_data":7297188,"kb_used_omap":724,"kb_used_meta":2199211,"kb_avail":84874708,"statfs":{"total":96636764160,"available":86911700992,"internally_reserved":0,"allocated":7472320512,"data_stored":8591197817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":2251992690},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.58799999999999997,"15min":0.44600000000000001},"min":{"1min":0.16,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":1.5309999999999999,"5min":5.8899999999999997,"15min":6.383},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.28299999999999997,"5min":0.31,"15min":0.435},"min":{"1min":0.19800000000000001,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":1.9350000000000001,"15min":6.0289999999999999},"last":0.32900000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.42399999999999999,"15min":0.501},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":4.0510000000000002,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.42299999999999999,"15min":0.62},"min":{"1min":0.27600000000000002,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":1.071,"5min":2.9929999999999999,"15min":26.596},"last":0.55900000000000005}]}]},{"osd":1,"up_from":8,"seq":34359739175,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4591988,"kb_used_data":3582252,"kb_used_omap":359,"kb_used_meta":1009368,"kb_avail":89779852,"statfs":{"total":96636764160,"available":91934568448,"internally_reserved":0,"allocated":3668226048,"data_stored":4214452217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1033592866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,4,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.376,"15min":0.46800000000000003},"min":{"1min":0.16800000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.52600000000000002,"5min":2.6320000000000001,"15min":13.757},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.21099999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9550000000000001,"5min":5.1459999999999999,"15min":13.183999999999999},"last":0.20300000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.501,"15min":0.55500000000000005},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.409,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.377,"15min":0.55100000000000005},"min":{"1min":0.23799999999999999,"5min":0.19800000000000001,"15min":0.193},"max":{"1min":1.3009999999999999,"5min":2.6640000000000001,"15min":19.521999999999998},"last":0.33400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3804487680,"data_stored":4376944640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3666972672,"data_stored":4213850131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7471661056,"data_stored":8590995475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:12:00.741 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:00.742 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:00.742 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:00.742 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:00.743 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:12:20.744 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:12:21.200 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:12:21.201 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:12:21.264 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2099,"stamp":"2023-10-21T05:12:19.642604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27474754792,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1853003,"num_read_kb":7234365,"num_write":1832145,"num_write_kb":7327559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76057,"ondisk_log_size":76057,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18635876,"kb_used_data":14652300,"kb_used_omap":1457,"kb_used_meta":3982094,"kb_avail":264479644,"statfs":{"total":289910292480,"available":270827155456,"internally_reserved":0,"allocated":15003955200,"data_stored":17240949507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":4077664452},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,2,4,7,8,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13000704,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3569,"num_read_kb":14248,"num_write":3551,"num_write_kb":14204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005192"},"pg_stats":[{"pgid":"2.7","version":"80'9117","reported_seq":12584,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:16.868043+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:12:16.868043+0000","last_peered":"2023-10-21T05:12:16.868043+0000","last_clean":"2023-10-21T05:12:16.868043+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:12:16.868043+0000","last_undegraded":"2023-10-21T05:12:16.868043+0000","last_fullsized":"2023-10-21T05:12:16.868043+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9117,"log_dups_size":0,"ondisk_log_size":9117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":996,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:40.354694+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:11:40.354694+0000","last_peered":"2023-10-21T05:11:40.354694+0000","last_clean":"2023-10-21T05:11:40.354694+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:11:40.354694+0000","last_undegraded":"2023-10-21T05:11:40.354694+0000","last_fullsized":"2023-10-21T05:11:40.354694+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"80'15047","reported_seq":20663,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:16.868341+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:12:16.868341+0000","last_peered":"2023-10-21T05:12:16.868341+0000","last_clean":"2023-10-21T05:12:16.868341+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:12:16.868341+0000","last_undegraded":"2023-10-21T05:12:16.868341+0000","last_fullsized":"2023-10-21T05:12:16.868341+0000","mapping_epoch":12,"log_start":"75'5000","ondisk_log_start":"75'5000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"80'3119","reported_seq":4869,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:16.471574+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:12:16.471574+0000","last_peered":"2023-10-21T05:12:16.471574+0000","last_clean":"2023-10-21T05:12:16.471574+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:12:16.471574+0000","last_undegraded":"2023-10-21T05:12:16.471574+0000","last_fullsized":"2023-10-21T05:12:16.471574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3119,"log_dups_size":0,"ondisk_log_size":3119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"80'462018","reported_seq":922343,"reported_epoch":80,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:17.288387+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:12:17.288387+0000","last_peered":"2023-10-21T05:12:17.288387+0000","last_clean":"2023-10-21T05:12:17.288387+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:12:17.288387+0000","last_undegraded":"2023-10-21T05:12:17.288387+0000","last_fullsized":"2023-10-21T05:12:17.288387+0000","mapping_epoch":15,"log_start":"79'452014","ondisk_log_start":"79'452014","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":549,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 70s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6905384960,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":461894,"num_read_kb":1817628,"num_write":459873,"num_write_kb":1839492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"80'15270","reported_seq":21108,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:16.868366+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:12:16.868366+0000","last_peered":"2023-10-21T05:12:16.868366+0000","last_clean":"2023-10-21T05:12:16.868366+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:12:16.868366+0000","last_undegraded":"2023-10-21T05:12:16.868366+0000","last_fullsized":"2023-10-21T05:12:16.868366+0000","mapping_epoch":12,"log_start":"75'5200","ondisk_log_start":"75'5200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"80'456888","reported_seq":910184,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:17.231479+0000","last_change":"2023-10-21T05:12:13.947457+0000","last_active":"2023-10-21T05:12:17.231479+0000","last_peered":"2023-10-21T05:12:17.231479+0000","last_clean":"2023-10-21T05:12:17.231479+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:12:17.231479+0000","last_undegraded":"2023-10-21T05:12:17.231479+0000","last_fullsized":"2023-10-21T05:12:17.231479+0000","mapping_epoch":15,"log_start":"79'446816","ondisk_log_start":"79'446816","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6811521024,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":454851,"num_read_kb":1789156,"num_write":455133,"num_write_kb":1820532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6093,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:17.045613+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:12:17.045613+0000","last_peered":"2023-10-21T05:12:17.045613+0000","last_clean":"2023-10-21T05:12:17.045613+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:12:17.045613+0000","last_undegraded":"2023-10-21T05:12:17.045613+0000","last_fullsized":"2023-10-21T05:12:17.045613+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"80'474849","reported_seq":946254,"reported_epoch":80,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:17.284411+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:12:17.284411+0000","last_peered":"2023-10-21T05:12:17.284411+0000","last_clean":"2023-10-21T05:12:17.284411+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:12:17.284411+0000","last_undegraded":"2023-10-21T05:12:17.284411+0000","last_fullsized":"2023-10-21T05:12:17.284411+0000","mapping_epoch":15,"log_start":"79'464823","ondisk_log_start":"79'464823","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":547,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 71s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7099936768,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":473035,"num_read_kb":1861420,"num_write":472640,"num_write_kb":1890560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":154,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:27.148690+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:11:27.148690+0000","last_peered":"2023-10-21T05:11:27.148690+0000","last_clean":"2023-10-21T05:11:27.148690+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:11:27.148690+0000","last_undegraded":"2023-10-21T05:11:27.148690+0000","last_fullsized":"2023-10-21T05:11:27.148690+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"80'445775","reported_seq":889324,"reported_epoch":80,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:17.239198+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:12:17.239198+0000","last_peered":"2023-10-21T05:12:17.239198+0000","last_clean":"2023-10-21T05:12:17.239198+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:12:17.239198+0000","last_undegraded":"2023-10-21T05:12:17.239198+0000","last_fullsized":"2023-10-21T05:12:17.239198+0000","mapping_epoch":15,"log_start":"79'435716","ondisk_log_start":"79'435716","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":529,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 67s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6657318931,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":445058,"num_read_kb":1750684,"num_write":443693,"num_write_kb":1774766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"80'3097","reported_seq":4382,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:16.465335+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:12:16.465335+0000","last_peered":"2023-10-21T05:12:16.465335+0000","last_clean":"2023-10-21T05:12:16.465335+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:12:16.465335+0000","last_undegraded":"2023-10-21T05:12:16.465335+0000","last_fullsized":"2023-10-21T05:12:16.465335+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3097,"log_dups_size":0,"ondisk_log_size":3097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":251,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:22.038918+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:11:22.038918+0000","last_peered":"2023-10-21T05:11:22.038918+0000","last_clean":"2023-10-21T05:11:22.038918+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:11:22.038918+0000","last_undegraded":"2023-10-21T05:11:22.038918+0000","last_fullsized":"2023-10-21T05:11:22.038918+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27474161683,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1834838,"num_read_kb":7218888,"num_write":1831339,"num_write_kb":7325350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15002501120,"data_stored":17241169958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40161,"ondisk_log_size":40161,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":35797,"ondisk_log_size":35797,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739181,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4922144,"kb_used_data":3731032,"kb_used_omap":373,"kb_used_meta":1190730,"kb_avail":89449696,"statfs":{"total":96636764160,"available":91596488704,"internally_reserved":0,"allocated":3820576768,"data_stored":4392467601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1219308080},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,0,2,3,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.378,"15min":0.41399999999999998},"min":{"1min":0.17000000000000001,"5min":0.14599999999999999,"15min":0.14299999999999999},"max":{"1min":2.0960000000000001,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.45100000000000001,"15min":0.42599999999999999},"min":{"1min":0.193,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":1.147,"5min":5.0060000000000002,"15min":5.9349999999999996},"last":0.20999999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51900000000000002,"15min":0.58599999999999997},"min":{"1min":0.217,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":2.1219999999999999,"5min":5.4100000000000001,"15min":14.318},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56399999999999995,"15min":0.64100000000000001},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.3050000000000002,"5min":5.4450000000000003,"15min":14.343},"last":0.54500000000000004}]}]},{"osd":2,"up_from":8,"seq":34359739180,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9051472,"kb_used_data":7324744,"kb_used_omap":724,"kb_used_meta":1725995,"kb_avail":85320368,"statfs":{"total":96636764160,"available":87368056832,"internally_reserved":0,"allocated":7500537856,"data_stored":8619415161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":1767419506},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.58799999999999997,"15min":0.44600000000000001},"min":{"1min":0.16,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":1.5309999999999999,"5min":5.8899999999999997,"15min":6.383},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.28299999999999997,"5min":0.31,"15min":0.435},"min":{"1min":0.19800000000000001,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":1.9350000000000001,"15min":6.0289999999999999},"last":0.249}]},{"osd":1,"last update":"Sat Oct 21 05:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.42399999999999999,"15min":0.501},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":4.0510000000000002,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.376},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.42299999999999999,"15min":0.62},"min":{"1min":0.27600000000000002,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":1.071,"5min":2.9929999999999999,"15min":26.596},"last":0.35399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739179,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4662260,"kb_used_data":3596524,"kb_used_omap":359,"kb_used_meta":1065368,"kb_avail":89709580,"statfs":{"total":96636764160,"available":91862609920,"internally_reserved":0,"allocated":3682840576,"data_stored":4229066745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1090936866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.376,"15min":0.46800000000000003},"min":{"1min":0.16800000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.52600000000000002,"5min":2.6320000000000001,"15min":13.757},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.21099999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":4.9550000000000001,"5min":5.1459999999999999,"15min":13.183999999999999},"last":0.35799999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.501,"15min":0.55500000000000005},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.409,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.377,"15min":0.55100000000000005},"min":{"1min":0.23799999999999999,"5min":0.19800000000000001,"15min":0.193},"max":{"1min":1.3009999999999999,"5min":2.6640000000000001,"15min":19.521999999999998},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3819573248,"data_stored":4392030208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3681591296,"data_stored":4228468755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7501336576,"data_stored":8620670995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:12:21.266 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:21.267 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:21.267 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:21.267 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:21.268 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:12:41.269 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:12:41.791 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:12:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:12:41.826 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2110,"stamp":"2023-10-21T05:12:41.646221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27501784296,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1860564,"num_read_kb":7264517,"num_write":1839525,"num_write_kb":7357079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76159,"ondisk_log_size":76159,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18836132,"kb_used_data":14712396,"kb_used_omap":1457,"kb_used_meta":4122254,"kb_avail":264279388,"statfs":{"total":289910292480,"available":270622093312,"internally_reserved":0,"allocated":15065493504,"data_stored":17302487811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":4221188292},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,1,7,13,5,12,14,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13688832,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3863,"num_read_kb":15428,"num_write":3729,"num_write_kb":14916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001946"},"pg_stats":[{"pgid":"2.7","version":"80'9141","reported_seq":12618,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:32.228042+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:12:32.228042+0000","last_peered":"2023-10-21T05:12:32.228042+0000","last_clean":"2023-10-21T05:12:32.228042+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:12:32.228042+0000","last_undegraded":"2023-10-21T05:12:32.228042+0000","last_fullsized":"2023-10-21T05:12:32.228042+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9141,"log_dups_size":0,"ondisk_log_size":9141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":996,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:40.354694+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:11:40.354694+0000","last_peered":"2023-10-21T05:11:40.354694+0000","last_clean":"2023-10-21T05:11:40.354694+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:11:40.354694+0000","last_undegraded":"2023-10-21T05:11:40.354694+0000","last_fullsized":"2023-10-21T05:11:40.354694+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"80'15087","reported_seq":20718,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:32.228142+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:12:32.228142+0000","last_peered":"2023-10-21T05:12:32.228142+0000","last_clean":"2023-10-21T05:12:32.228142+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:12:32.228142+0000","last_undegraded":"2023-10-21T05:12:32.228142+0000","last_fullsized":"2023-10-21T05:12:32.228142+0000","mapping_epoch":12,"log_start":"75'5000","ondisk_log_start":"75'5000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"80'3127","reported_seq":4880,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:31.855402+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:12:31.855402+0000","last_peered":"2023-10-21T05:12:31.855402+0000","last_clean":"2023-10-21T05:12:31.855402+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:12:31.855402+0000","last_undegraded":"2023-10-21T05:12:31.855402+0000","last_fullsized":"2023-10-21T05:12:31.855402+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"80'464023","reported_seq":926270,"reported_epoch":80,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:37.428394+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:12:37.428394+0000","last_peered":"2023-10-21T05:12:37.428394+0000","last_clean":"2023-10-21T05:12:37.428394+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:12:37.428394+0000","last_undegraded":"2023-10-21T05:12:37.428394+0000","last_fullsized":"2023-10-21T05:12:37.428394+0000","mapping_epoch":15,"log_start":"79'454014","ondisk_log_start":"79'454014","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":690,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 90s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6912376832,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":463800,"num_read_kb":1825232,"num_write":461784,"num_write_kb":1847136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"80'15321","reported_seq":21179,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:37.348532+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:12:37.348532+0000","last_peered":"2023-10-21T05:12:37.348532+0000","last_clean":"2023-10-21T05:12:37.348532+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:12:37.348532+0000","last_undegraded":"2023-10-21T05:12:37.348532+0000","last_fullsized":"2023-10-21T05:12:37.348532+0000","mapping_epoch":12,"log_start":"75'5300","ondisk_log_start":"75'5300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"80'458696","reported_seq":913891,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:37.304891+0000","last_change":"2023-10-21T05:12:33.996682+0000","last_active":"2023-10-21T05:12:37.304891+0000","last_peered":"2023-10-21T05:12:37.304891+0000","last_clean":"2023-10-21T05:12:37.304891+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:12:37.304891+0000","last_undegraded":"2023-10-21T05:12:37.304891+0000","last_fullsized":"2023-10-21T05:12:37.304891+0000","mapping_epoch":15,"log_start":"79'448616","ondisk_log_start":"79'448616","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6818062336,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":456743,"num_read_kb":1796696,"num_write":456941,"num_write_kb":1827764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6113,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:37.356894+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:12:37.356894+0000","last_peered":"2023-10-21T05:12:37.356894+0000","last_clean":"2023-10-21T05:12:37.356894+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:12:37.356894+0000","last_undegraded":"2023-10-21T05:12:37.356894+0000","last_fullsized":"2023-10-21T05:12:37.356894+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"80'476820","reported_seq":950158,"reported_epoch":80,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:37.417413+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:12:37.417413+0000","last_peered":"2023-10-21T05:12:37.417413+0000","last_clean":"2023-10-21T05:12:37.417413+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:12:37.417413+0000","last_undegraded":"2023-10-21T05:12:37.417413+0000","last_fullsized":"2023-10-21T05:12:37.417413+0000","mapping_epoch":15,"log_start":"79'466723","ondisk_log_start":"79'466723","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":697,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 91s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7106871296,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":474952,"num_read_kb":1869072,"num_write":474512,"num_write_kb":1898048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":154,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:27.148690+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:11:27.148690+0000","last_peered":"2023-10-21T05:11:27.148690+0000","last_clean":"2023-10-21T05:11:27.148690+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:11:27.148690+0000","last_undegraded":"2023-10-21T05:11:27.148690+0000","last_fullsized":"2023-10-21T05:11:27.148690+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"80'447662","reported_seq":893069,"reported_epoch":80,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:37.302208+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:12:37.302208+0000","last_peered":"2023-10-21T05:12:37.302208+0000","last_clean":"2023-10-21T05:12:37.302208+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:12:37.302208+0000","last_undegraded":"2023-10-21T05:12:37.302208+0000","last_fullsized":"2023-10-21T05:12:37.302208+0000","mapping_epoch":15,"log_start":"79'437616","ondisk_log_start":"79'437616","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":682,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 87s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6663880723,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":446904,"num_read_kb":1758040,"num_write":445482,"num_write_kb":1781922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"80'3105","reported_seq":4393,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:31.873608+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:12:31.873608+0000","last_peered":"2023-10-21T05:12:31.873608+0000","last_clean":"2023-10-21T05:12:31.873608+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:12:31.873608+0000","last_undegraded":"2023-10-21T05:12:31.873608+0000","last_fullsized":"2023-10-21T05:12:31.873608+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3105,"log_dups_size":0,"ondisk_log_size":3105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":251,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-10-21T05:11:22.038918+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:11:22.038918+0000","last_peered":"2023-10-21T05:11:22.038918+0000","last_clean":"2023-10-21T05:11:22.038918+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:11:22.038918+0000","last_undegraded":"2023-10-21T05:11:22.038918+0000","last_fullsized":"2023-10-21T05:11:22.038918+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27501191187,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1842399,"num_read_kb":7249040,"num_write":1838719,"num_write_kb":7354870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15062933504,"data_stored":17301602342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40232,"ondisk_log_size":40232,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":35828,"ondisk_log_size":35828,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739185,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4973348,"kb_used_data":3746140,"kb_used_omap":373,"kb_used_meta":1226826,"kb_avail":89398492,"statfs":{"total":96636764160,"available":91544055808,"internally_reserved":0,"allocated":3836047360,"data_stored":4407938193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1256270384},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,4,6,0,7,11,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.378,"15min":0.41399999999999998},"min":{"1min":0.17000000000000001,"5min":0.14599999999999999,"15min":0.14299999999999999},"max":{"1min":2.0960000000000001,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.17799999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.45100000000000001,"15min":0.42599999999999999},"min":{"1min":0.193,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":1.147,"5min":5.0060000000000002,"15min":5.9349999999999996},"last":0.22500000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51900000000000002,"15min":0.58599999999999997},"min":{"1min":0.217,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":2.1219999999999999,"5min":5.4100000000000001,"15min":14.318},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56399999999999995,"15min":0.64100000000000001},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.3050000000000002,"5min":5.4450000000000003,"15min":14.343},"last":0.32500000000000001}]}]},{"osd":2,"up_from":8,"seq":34359739184,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9168116,"kb_used_data":7355372,"kb_used_omap":724,"kb_used_meta":1812011,"kb_avail":85203724,"statfs":{"total":96636764160,"available":87248613376,"internally_reserved":0,"allocated":7531900928,"data_stored":8650778233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":1855499890},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.58799999999999997,"15min":0.44600000000000001},"min":{"1min":0.16,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":1.5309999999999999,"5min":5.8899999999999997,"15min":6.383},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.28299999999999997,"5min":0.31,"15min":0.435},"min":{"1min":0.19800000000000001,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":1.9350000000000001,"15min":6.0289999999999999},"last":0.40600000000000003}]},{"osd":1,"last update":"Sat Oct 21 05:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.42399999999999999,"15min":0.501},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":4.0510000000000002,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.42299999999999999,"15min":0.62},"min":{"1min":0.27600000000000002,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":1.071,"5min":2.9929999999999999,"15min":26.596},"last":0.60999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739183,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4694668,"kb_used_data":3610884,"kb_used_omap":359,"kb_used_meta":1083416,"kb_avail":89677172,"statfs":{"total":96636764160,"available":91829424128,"internally_reserved":0,"allocated":3697545216,"data_stored":4243771385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1109418018},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,6,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:21 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39300000000000002,"15min":0.42499999999999999},"min":{"1min":0.191,"5min":0.16800000000000001,"15min":0.16400000000000001},"max":{"1min":3.1339999999999999,"5min":3.1339999999999999,"15min":9.218},"last":0.219},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57499999999999996,"15min":0.52600000000000002},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":6.2839999999999998},"last":0.33200000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:12:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.47799999999999998,"15min":0.55800000000000005},"min":{"1min":0.22600000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.998,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40200000000000002,"15min":0.51600000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.193},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":19.521999999999998},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3835068416,"data_stored":4407525376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3696320512,"data_stored":4243197971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7531544576,"data_stored":8650878995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:12:41.828 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:41.828 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:41.828 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:41.829 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:41.829 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:12:47.512 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:12:47.509+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T05:12:47.513 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:12:47.509+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130c30 2023-10-21T05:12:47.513 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:12:47.509+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T05:12:47.513 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:12:47.509+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T05:12:47.513 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:12:47.509+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:13:01.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:13:02.362 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:13:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:13:02.398 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2121,"stamp":"2023-10-21T05:13:01.649398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27528314088,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1867916,"num_read_kb":7293825,"num_write":1846766,"num_write_kb":7386043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76200,"ondisk_log_size":76200,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18996852,"kb_used_data":14770396,"kb_used_omap":1457,"kb_used_meta":4224974,"kb_avail":264118668,"statfs":{"total":289910292480,"available":270457516032,"internally_reserved":0,"allocated":15124885504,"data_stored":17361866866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":4326373572},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,5,5,14,17,16,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13103104,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3752,"num_read_kb":14972,"num_write":3587,"num_write_kb":14348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.258632"},"pg_stats":[{"pgid":"2.7","version":"81'9165","reported_seq":12655,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:12:54.618954+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:12:54.618954+0000","last_peered":"2023-10-21T05:12:54.618954+0000","last_clean":"2023-10-21T05:12:54.618954+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:12:54.618954+0000","last_undegraded":"2023-10-21T05:12:54.618954+0000","last_fullsized":"2023-10-21T05:12:54.618954+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9165,"log_dups_size":0,"ondisk_log_size":9165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":997,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:17.243907+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:12:17.243907+0000","last_peered":"2023-10-21T05:12:17.243907+0000","last_clean":"2023-10-21T05:12:17.243907+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:12:17.243907+0000","last_undegraded":"2023-10-21T05:12:17.243907+0000","last_fullsized":"2023-10-21T05:12:17.243907+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"81'15127","reported_seq":20779,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:12:54.624218+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:12:54.624218+0000","last_peered":"2023-10-21T05:12:54.624218+0000","last_clean":"2023-10-21T05:12:54.624218+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:12:54.624218+0000","last_undegraded":"2023-10-21T05:12:54.624218+0000","last_fullsized":"2023-10-21T05:12:54.624218+0000","mapping_epoch":12,"log_start":"75'5100","ondisk_log_start":"75'5100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"81'3135","reported_seq":4893,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:12:53.310609+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:12:53.310609+0000","last_peered":"2023-10-21T05:12:53.310609+0000","last_clean":"2023-10-21T05:12:53.310609+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:12:53.310609+0000","last_undegraded":"2023-10-21T05:12:53.310609+0000","last_fullsized":"2023-10-21T05:12:53.310609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"81'466028","reported_seq":930124,"reported_epoch":81,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:57.549001+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:12:57.549001+0000","last_peered":"2023-10-21T05:12:57.549001+0000","last_clean":"2023-10-21T05:12:57.549001+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:12:57.549001+0000","last_undegraded":"2023-10-21T05:12:57.549001+0000","last_fullsized":"2023-10-21T05:12:57.549001+0000","mapping_epoch":15,"log_start":"79'456014","ondisk_log_start":"79'456014","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":908,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 111s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6919217152,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":465631,"num_read_kb":1832552,"num_write":463654,"num_write_kb":1854616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"81'15366","reported_seq":21240,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:12:52.843148+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:12:52.843148+0000","last_peered":"2023-10-21T05:12:52.843148+0000","last_clean":"2023-10-21T05:12:52.843148+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:12:52.843148+0000","last_undegraded":"2023-10-21T05:12:52.843148+0000","last_fullsized":"2023-10-21T05:12:52.843148+0000","mapping_epoch":12,"log_start":"75'5300","ondisk_log_start":"75'5300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"81'460510","reported_seq":917610,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:12:57.364624+0000","last_change":"2023-10-21T05:12:57.151851+0000","last_active":"2023-10-21T05:12:57.364624+0000","last_peered":"2023-10-21T05:12:57.364624+0000","last_clean":"2023-10-21T05:12:57.364624+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:12:57.364624+0000","last_undegraded":"2023-10-21T05:12:57.364624+0000","last_fullsized":"2023-10-21T05:12:57.364624+0000","mapping_epoch":15,"log_start":"79'450416","ondisk_log_start":"79'450416","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6824726528,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":458645,"num_read_kb":1804268,"num_write":458755,"num_write_kb":1835020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6129,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:12:52.710874+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:12:52.710874+0000","last_peered":"2023-10-21T05:12:52.710874+0000","last_clean":"2023-10-21T05:12:52.710874+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:12:52.710874+0000","last_undegraded":"2023-10-21T05:12:52.710874+0000","last_fullsized":"2023-10-21T05:12:52.710874+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"81'478789","reported_seq":953961,"reported_epoch":81,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:57.536974+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:12:57.536974+0000","last_peered":"2023-10-21T05:12:57.536974+0000","last_clean":"2023-10-21T05:12:57.536974+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:12:57.536974+0000","last_undegraded":"2023-10-21T05:12:57.536974+0000","last_fullsized":"2023-10-21T05:12:57.536974+0000","mapping_epoch":15,"log_start":"79'468723","ondisk_log_start":"79'468723","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":925,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 111s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7113609216,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":476770,"num_read_kb":1876300,"num_write":476332,"num_write_kb":1905328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":155,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:40.365625+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:11:40.365625+0000","last_peered":"2023-10-21T05:11:40.365625+0000","last_clean":"2023-10-21T05:11:40.365625+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:11:40.365625+0000","last_undegraded":"2023-10-21T05:11:40.365625+0000","last_fullsized":"2023-10-21T05:11:40.365625+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"81'449491","reported_seq":896709,"reported_epoch":81,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:57.315833+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:12:57.315833+0000","last_peered":"2023-10-21T05:12:57.315833+0000","last_clean":"2023-10-21T05:12:57.315833+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:12:57.315833+0000","last_undegraded":"2023-10-21T05:12:57.315833+0000","last_fullsized":"2023-10-21T05:12:57.315833+0000","mapping_epoch":15,"log_start":"79'439417","ondisk_log_start":"79'439417","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":823,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 107s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6670168083,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":448705,"num_read_kb":1765228,"num_write":447219,"num_write_kb":1788870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"81'3113","reported_seq":4406,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:12:53.333148+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:12:53.333148+0000","last_peered":"2023-10-21T05:12:53.333148+0000","last_clean":"2023-10-21T05:12:53.333148+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:12:53.333148+0000","last_undegraded":"2023-10-21T05:12:53.333148+0000","last_fullsized":"2023-10-21T05:12:53.333148+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":252,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:40.366711+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:11:40.366711+0000","last_peered":"2023-10-21T05:11:40.366711+0000","last_clean":"2023-10-21T05:11:40.366711+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:11:40.366711+0000","last_undegraded":"2023-10-21T05:11:40.366711+0000","last_fullsized":"2023-10-21T05:11:40.366711+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27527720979,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1849751,"num_read_kb":7278348,"num_write":1845960,"num_write_kb":7383834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15122280448,"data_stored":17360949286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40248,"ondisk_log_size":40248,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":35853,"ondisk_log_size":35853,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739189,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5024184,"kb_used_data":3760880,"kb_used_omap":373,"kb_used_meta":1262922,"kb_avail":89347656,"statfs":{"total":96636764160,"available":91491999744,"internally_reserved":0,"allocated":3851141120,"data_stored":4423027638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1293232688},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,0,7,12,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.378,"15min":0.41399999999999998},"min":{"1min":0.17000000000000001,"5min":0.14599999999999999,"15min":0.14299999999999999},"max":{"1min":2.0960000000000001,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.45100000000000001,"15min":0.42599999999999999},"min":{"1min":0.193,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":1.147,"5min":5.0060000000000002,"15min":5.9349999999999996},"last":0.27500000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51900000000000002,"15min":0.58599999999999997},"min":{"1min":0.217,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":2.1219999999999999,"5min":5.4100000000000001,"15min":14.318},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56399999999999995,"15min":0.64100000000000001},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":2.3050000000000002,"5min":5.4450000000000003,"15min":14.343},"last":0.29499999999999998}]}]},{"osd":2,"up_from":8,"seq":34359739188,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9245604,"kb_used_data":7384284,"kb_used_omap":724,"kb_used_meta":1860587,"kb_avail":85126236,"statfs":{"total":96636764160,"available":87169265664,"internally_reserved":0,"allocated":7561506816,"data_stored":8680379806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":1905241714},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:48 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.59999999999999998,"15min":0.45000000000000001},"min":{"1min":0.16700000000000001,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":0.81899999999999995,"5min":5.8899999999999997,"15min":6.383},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.315,"5min":0.28799999999999998,"15min":0.437},"min":{"1min":0.19700000000000001,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":6.0289999999999999},"last":0.59499999999999997}]},{"osd":1,"last update":"Sat Oct 21 05:12:48 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.42599999999999999,"15min":0.499},"min":{"1min":0.252,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":0.52800000000000002,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.40600000000000003,"15min":0.624},"min":{"1min":0.23599999999999999,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":0.63300000000000001,"5min":2.1280000000000001,"15min":26.596},"last":0.83299999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739187,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4727064,"kb_used_data":3625232,"kb_used_omap":359,"kb_used_meta":1101464,"kb_avail":89644776,"statfs":{"total":96636764160,"available":91796250624,"internally_reserved":0,"allocated":3712237568,"data_stored":4258459422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1127899170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,5,5,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:21 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39300000000000002,"15min":0.42499999999999999},"min":{"1min":0.191,"5min":0.16800000000000001,"15min":0.16400000000000001},"max":{"1min":3.1339999999999999,"5min":3.1339999999999999,"15min":9.218},"last":0.255},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57499999999999996,"15min":0.52600000000000002},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":6.2839999999999998},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:12:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.47799999999999998,"15min":0.55800000000000005},"min":{"1min":0.22600000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.998,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40200000000000002,"15min":0.51600000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.193},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":19.521999999999998},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3850186752,"data_stored":4422643712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3710873600,"data_stored":4257751059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7561220096,"data_stored":8680554515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:13:02.400 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:02.400 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:02.400 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:02.401 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:02.401 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:13:22.403 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:13:23.633 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:13:23.634 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:13:23.668 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2131,"stamp":"2023-10-21T05:13:21.652725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27553934568,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1875032,"num_read_kb":7322121,"num_write":1853807,"num_write_kb":7414207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76333,"ondisk_log_size":76333,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19248288,"kb_used_data":14826248,"kb_used_omap":1457,"kb_used_meta":4420558,"kb_avail":263867232,"statfs":{"total":289910292480,"available":270200045568,"internally_reserved":0,"allocated":15182077952,"data_stored":17419059314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":4526651588},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,6,5,8,15,17,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13213696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3647,"num_read_kb":14488,"num_write":3602,"num_write_kb":14408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001999"},"pg_stats":[{"pgid":"2.7","version":"81'9192","reported_seq":12694,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:13.190121+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:13:13.190121+0000","last_peered":"2023-10-21T05:13:13.190121+0000","last_clean":"2023-10-21T05:13:13.190121+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:13:13.190121+0000","last_undegraded":"2023-10-21T05:13:13.190121+0000","last_fullsized":"2023-10-21T05:13:13.190121+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9192,"log_dups_size":0,"ondisk_log_size":9192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":997,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:17.243907+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:12:17.243907+0000","last_peered":"2023-10-21T05:12:17.243907+0000","last_clean":"2023-10-21T05:12:17.243907+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:12:17.243907+0000","last_undegraded":"2023-10-21T05:12:17.243907+0000","last_fullsized":"2023-10-21T05:12:17.243907+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"81'15172","reported_seq":20844,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:13.188398+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:13:13.188398+0000","last_peered":"2023-10-21T05:13:13.188398+0000","last_clean":"2023-10-21T05:13:13.188398+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:13:13.188398+0000","last_undegraded":"2023-10-21T05:13:13.188398+0000","last_fullsized":"2023-10-21T05:13:13.188398+0000","mapping_epoch":12,"log_start":"75'5100","ondisk_log_start":"75'5100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"81'3144","reported_seq":4906,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:08.279411+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:13:08.279411+0000","last_peered":"2023-10-21T05:13:08.279411+0000","last_clean":"2023-10-21T05:13:08.279411+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:13:08.279411+0000","last_undegraded":"2023-10-21T05:13:08.279411+0000","last_fullsized":"2023-10-21T05:13:08.279411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3144,"log_dups_size":0,"ondisk_log_size":3144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"81'467881","reported_seq":933772,"reported_epoch":81,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:17.666479+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:13:17.666479+0000","last_peered":"2023-10-21T05:13:17.666479+0000","last_clean":"2023-10-21T05:13:17.666479+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:13:17.666479+0000","last_undegraded":"2023-10-21T05:13:17.666479+0000","last_fullsized":"2023-10-21T05:13:17.666479+0000","mapping_epoch":15,"log_start":"79'457814","ondisk_log_start":"79'457814","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1146,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 131s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6925299712,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":467408,"num_read_kb":1839620,"num_write":465354,"num_write_kb":1861416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"81'15471","reported_seq":21365,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:16.702506+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:13:16.702506+0000","last_peered":"2023-10-21T05:13:16.702506+0000","last_clean":"2023-10-21T05:13:16.702506+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:13:16.702506+0000","last_undegraded":"2023-10-21T05:13:16.702506+0000","last_fullsized":"2023-10-21T05:13:16.702506+0000","mapping_epoch":12,"log_start":"75'5400","ondisk_log_start":"75'5400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"81'462364","reported_seq":921274,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:17.453246+0000","last_change":"2023-10-21T05:13:15.255204+0000","last_active":"2023-10-21T05:13:17.453246+0000","last_peered":"2023-10-21T05:13:17.453246+0000","last_clean":"2023-10-21T05:13:17.453246+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:13:17.453246+0000","last_undegraded":"2023-10-21T05:13:17.453246+0000","last_fullsized":"2023-10-21T05:13:17.453246+0000","mapping_epoch":15,"log_start":"79'452316","ondisk_log_start":"79'452316","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6831427584,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":460447,"num_read_kb":1811424,"num_write":460609,"num_write_kb":1842436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6149,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:13.188613+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:13:13.188613+0000","last_peered":"2023-10-21T05:13:13.188613+0000","last_clean":"2023-10-21T05:13:13.188613+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:13:13.188613+0000","last_undegraded":"2023-10-21T05:13:13.188613+0000","last_fullsized":"2023-10-21T05:13:13.188613+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"81'480710","reported_seq":957734,"reported_epoch":81,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:17.658305+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:13:17.658305+0000","last_peered":"2023-10-21T05:13:17.658305+0000","last_clean":"2023-10-21T05:13:17.658305+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:13:17.658305+0000","last_undegraded":"2023-10-21T05:13:17.658305+0000","last_fullsized":"2023-10-21T05:13:17.658305+0000","mapping_epoch":15,"log_start":"79'470623","ondisk_log_start":"79'470623","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1099,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 132s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7120285696,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":478605,"num_read_kb":1883616,"num_write":478140,"num_write_kb":1912560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":155,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:40.365625+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:11:40.365625+0000","last_peered":"2023-10-21T05:11:40.365625+0000","last_clean":"2023-10-21T05:11:40.365625+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:11:40.365625+0000","last_undegraded":"2023-10-21T05:11:40.365625+0000","last_fullsized":"2023-10-21T05:11:40.365625+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"81'451301","reported_seq":900249,"reported_epoch":81,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:17.443445+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:13:17.443445+0000","last_peered":"2023-10-21T05:13:17.443445+0000","last_clean":"2023-10-21T05:13:17.443445+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:13:17.443445+0000","last_undegraded":"2023-10-21T05:13:17.443445+0000","last_fullsized":"2023-10-21T05:13:17.443445+0000","mapping_epoch":15,"log_start":"79'441217","ondisk_log_start":"79'441217","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1032,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 127s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6676328467,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":450407,"num_read_kb":1771984,"num_write":448898,"num_write_kb":1795586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"81'3122","reported_seq":4419,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:08.329586+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:13:08.329586+0000","last_peered":"2023-10-21T05:13:08.329586+0000","last_clean":"2023-10-21T05:13:08.329586+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:13:08.329586+0000","last_undegraded":"2023-10-21T05:13:08.329586+0000","last_fullsized":"2023-10-21T05:13:08.329586+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3122,"log_dups_size":0,"ondisk_log_size":3122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":252,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:11:40.366711+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:11:40.366711+0000","last_peered":"2023-10-21T05:11:40.366711+0000","last_clean":"2023-10-21T05:11:40.366711+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:11:40.366711+0000","last_undegraded":"2023-10-21T05:11:40.366711+0000","last_fullsized":"2023-10-21T05:11:40.366711+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27553341459,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1856867,"num_read_kb":7306644,"num_write":1853001,"num_write_kb":7411998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15180021760,"data_stored":17418690598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40286,"ondisk_log_size":40286,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":35948,"ondisk_log_size":35948,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739193,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5092332,"kb_used_data":3774884,"kb_used_omap":373,"kb_used_meta":1317066,"kb_avail":89279508,"statfs":{"total":96636764160,"available":91422216192,"internally_reserved":0,"allocated":3865481216,"data_stored":4437367734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1348676144},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,3,10,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.33600000000000002,"15min":0.40999999999999998},"min":{"1min":0.16700000000000001,"5min":0.14599999999999999,"15min":0.14299999999999999},"max":{"1min":0.47399999999999998,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.42899999999999999,"15min":0.41499999999999998},"min":{"1min":0.20999999999999999,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":0.624,"5min":5.0060000000000002,"15min":5.9349999999999996},"last":0.28799999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.52400000000000002,"15min":0.58299999999999996},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":0.57699999999999996,"5min":5.4100000000000001,"15min":14.318},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.56599999999999995,"15min":0.63900000000000001},"min":{"1min":0.27100000000000002,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":0.60699999999999998,"5min":5.4450000000000003,"15min":14.343},"last":0.434}]}]},{"osd":2,"up_from":8,"seq":34359739192,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9378712,"kb_used_data":7412048,"kb_used_omap":724,"kb_used_meta":1965931,"kb_avail":84993128,"statfs":{"total":96636764160,"available":87032963072,"internally_reserved":0,"allocated":7589937152,"data_stored":8708810142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":2013113970},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:48 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.59999999999999998,"15min":0.45000000000000001},"min":{"1min":0.16700000000000001,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":0.81899999999999995,"5min":5.8899999999999997,"15min":6.383},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.315,"5min":0.28799999999999998,"15min":0.437},"min":{"1min":0.19700000000000001,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":6.0289999999999999},"last":0.41599999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:12:48 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.42599999999999999,"15min":0.499},"min":{"1min":0.252,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":0.52800000000000002,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.40600000000000003,"15min":0.624},"min":{"1min":0.23599999999999999,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":0.63300000000000001,"5min":2.1280000000000001,"15min":26.596},"last":0.45800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739191,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4777244,"kb_used_data":3639316,"kb_used_omap":359,"kb_used_meta":1137560,"kb_avail":89594596,"statfs":{"total":96636764160,"available":91744866304,"internally_reserved":0,"allocated":3726659584,"data_stored":4272881438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1164861474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,4,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:21 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39300000000000002,"15min":0.42499999999999999},"min":{"1min":0.191,"5min":0.16800000000000001,"15min":0.16400000000000001},"max":{"1min":3.1339999999999999,"5min":3.1339999999999999,"15min":9.218},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57499999999999996,"15min":0.52600000000000002},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":6.2839999999999998},"last":0.38900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:12:21 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.47799999999999998,"15min":0.55800000000000005},"min":{"1min":0.22600000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.998,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40200000000000002,"15min":0.51600000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.193},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":19.521999999999998},"last":0.311}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3864571904,"data_stored":4437028864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3725340672,"data_stored":4272218131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7590109184,"data_stored":8709443603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:13:23.670 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:23.671 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:23.671 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:23.671 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:23.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:13:24.159 INFO:teuthology.orchestra.run.smithi119.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-21T05:13:24.187 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:13:24.764 INFO:teuthology.orchestra.run.smithi119.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:13:24.800 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:13:25.470 INFO:teuthology.orchestra.run.smithi119.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-21T05:13:25.503 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:13:26.043 INFO:teuthology.orchestra.run.smithi119.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:13:26.074 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:13:46.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:13:46.546 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:13:46.547 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:13:46.574 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2143,"stamp":"2023-10-21T05:13:45.658139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27586903272,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1884168,"num_read_kb":7358465,"num_write":1862786,"num_write_kb":7450123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76385,"ondisk_log_size":76385,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19468672,"kb_used_data":14898088,"kb_used_omap":1457,"kb_used_meta":4569102,"kb_avail":263646848,"statfs":{"total":289910292480,"available":269974372352,"internally_reserved":0,"allocated":15255642112,"data_stored":17492623474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":4678760644},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,7,9,12,14,11,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13398016,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3697,"num_read_kb":14716,"num_write":3628,"num_write_kb":14512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001939"},"pg_stats":[{"pgid":"2.7","version":"81'9252","reported_seq":12769,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:41.882970+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:13:41.882970+0000","last_peered":"2023-10-21T05:13:41.882970+0000","last_clean":"2023-10-21T05:13:41.882970+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:13:41.882970+0000","last_undegraded":"2023-10-21T05:13:41.882970+0000","last_fullsized":"2023-10-21T05:13:41.882970+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9252,"log_dups_size":0,"ondisk_log_size":9252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":997,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-10-21T05:12:17.243907+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:12:17.243907+0000","last_peered":"2023-10-21T05:12:17.243907+0000","last_clean":"2023-10-21T05:12:17.243907+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:12:17.243907+0000","last_undegraded":"2023-10-21T05:12:17.243907+0000","last_fullsized":"2023-10-21T05:12:17.243907+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"81'15272","reported_seq":20969,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:41.899445+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:13:41.899445+0000","last_peered":"2023-10-21T05:13:41.899445+0000","last_clean":"2023-10-21T05:13:41.899445+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:13:41.899445+0000","last_undegraded":"2023-10-21T05:13:41.899445+0000","last_fullsized":"2023-10-21T05:13:41.899445+0000","mapping_epoch":12,"log_start":"75'5200","ondisk_log_start":"75'5200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"81'3164","reported_seq":4931,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:39.899793+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:13:39.899793+0000","last_peered":"2023-10-21T05:13:39.899793+0000","last_clean":"2023-10-21T05:13:39.899793+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:13:39.899793+0000","last_undegraded":"2023-10-21T05:13:39.899793+0000","last_fullsized":"2023-10-21T05:13:39.899793+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3164,"log_dups_size":0,"ondisk_log_size":3164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"81'470282","reported_seq":938427,"reported_epoch":81,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:42.930939+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:13:42.930939+0000","last_peered":"2023-10-21T05:13:42.930939+0000","last_clean":"2023-10-21T05:13:42.930939+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:13:42.930939+0000","last_undegraded":"2023-10-21T05:13:42.930939+0000","last_fullsized":"2023-10-21T05:13:42.930939+0000","mapping_epoch":15,"log_start":"80'460214","ondisk_log_start":"80'460214","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1333,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 156s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6933729280,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":469640,"num_read_kb":1848512,"num_write":467641,"num_write_kb":1870564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"81'15566","reported_seq":21485,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:42.326547+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:13:42.326547+0000","last_peered":"2023-10-21T05:13:42.326547+0000","last_clean":"2023-10-21T05:13:42.326547+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:13:42.326547+0000","last_undegraded":"2023-10-21T05:13:42.326547+0000","last_fullsized":"2023-10-21T05:13:42.326547+0000","mapping_epoch":12,"log_start":"75'5500","ondisk_log_start":"75'5500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"81'464556","reported_seq":925816,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:42.552050+0000","last_change":"2023-10-21T05:13:40.340293+0000","last_active":"2023-10-21T05:13:42.552050+0000","last_peered":"2023-10-21T05:13:42.552050+0000","last_clean":"2023-10-21T05:13:42.552050+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:13:42.552050+0000","last_undegraded":"2023-10-21T05:13:42.552050+0000","last_fullsized":"2023-10-21T05:13:42.552050+0000","mapping_epoch":15,"log_start":"80'454516","ondisk_log_start":"80'454516","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6839410688,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":462790,"num_read_kb":1820720,"num_write":462801,"num_write_kb":1851204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6174,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:38.801061+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:13:38.801061+0000","last_peered":"2023-10-21T05:13:38.801061+0000","last_clean":"2023-10-21T05:13:38.801061+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:13:38.801061+0000","last_undegraded":"2023-10-21T05:13:38.801061+0000","last_fullsized":"2023-10-21T05:13:38.801061+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"81'483170","reported_seq":962486,"reported_epoch":81,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:42.943955+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:13:42.943955+0000","last_peered":"2023-10-21T05:13:42.943955+0000","last_clean":"2023-10-21T05:13:42.943955+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:13:42.943955+0000","last_undegraded":"2023-10-21T05:13:42.943955+0000","last_fullsized":"2023-10-21T05:13:42.943955+0000","mapping_epoch":15,"log_start":"80'473124","ondisk_log_start":"80'473124","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1320,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 157s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7128862720,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":480889,"num_read_kb":1892704,"num_write":480461,"num_write_kb":1921844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":156,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:12:50.395592+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:12:50.395592+0000","last_peered":"2023-10-21T05:12:50.395592+0000","last_clean":"2023-10-21T05:12:50.395592+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:12:50.395592+0000","last_undegraded":"2023-10-21T05:12:50.395592+0000","last_fullsized":"2023-10-21T05:12:50.395592+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"81'453606","reported_seq":904852,"reported_epoch":81,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:42.566632+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:13:42.566632+0000","last_peered":"2023-10-21T05:13:42.566632+0000","last_clean":"2023-10-21T05:13:42.566632+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:13:42.566632+0000","last_undegraded":"2023-10-21T05:13:42.566632+0000","last_fullsized":"2023-10-21T05:13:42.566632+0000","mapping_epoch":15,"log_start":"80'443517","ondisk_log_start":"80'443517","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1228,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 152s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6684307475,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":452684,"num_read_kb":1781052,"num_write":451077,"num_write_kb":1804302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"81'3142","reported_seq":4444,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:39.910178+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:13:39.910178+0000","last_peered":"2023-10-21T05:13:39.910178+0000","last_clean":"2023-10-21T05:13:39.910178+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:13:39.910178+0000","last_undegraded":"2023-10-21T05:13:39.910178+0000","last_fullsized":"2023-10-21T05:13:39.910178+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":253,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:12:50.398195+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:12:50.398195+0000","last_peered":"2023-10-21T05:12:50.398195+0000","last_clean":"2023-10-21T05:12:50.398195+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:12:50.398195+0000","last_undegraded":"2023-10-21T05:12:50.398195+0000","last_fullsized":"2023-10-21T05:12:50.398195+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27586310163,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1866003,"num_read_kb":7342988,"num_write":1861980,"num_write_kb":7447914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15253598208,"data_stored":17492267046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40243,"ondisk_log_size":40243,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36043,"ondisk_log_size":36043,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739198,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5132880,"kb_used_data":3793224,"kb_used_omap":373,"kb_used_meta":1339274,"kb_avail":89238960,"statfs":{"total":96636764160,"available":91380695040,"internally_reserved":0,"allocated":3884261376,"data_stored":4456147894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1371417136},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,2,9,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.33600000000000002,"15min":0.40999999999999998},"min":{"1min":0.16700000000000001,"5min":0.14599999999999999,"15min":0.14299999999999999},"max":{"1min":0.47399999999999998,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.42899999999999999,"15min":0.41499999999999998},"min":{"1min":0.20999999999999999,"5min":0.19,"15min":0.17699999999999999},"max":{"1min":0.624,"5min":5.0060000000000002,"15min":5.9349999999999996},"last":0.75900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.52400000000000002,"15min":0.58299999999999996},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":0.57699999999999996,"5min":5.4100000000000001,"15min":14.318},"last":0.307},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.56599999999999995,"15min":0.63900000000000001},"min":{"1min":0.27100000000000002,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":0.60699999999999998,"5min":5.4450000000000003,"15min":14.343},"last":0.34000000000000002}]}]},{"osd":2,"up_from":8,"seq":34359739197,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9486876,"kb_used_data":7448020,"kb_used_omap":724,"kb_used_meta":2038123,"kb_avail":84884964,"statfs":{"total":96636764160,"available":86922203136,"internally_reserved":0,"allocated":7626772480,"data_stored":8745645470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":2087038578},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:48 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.59999999999999998,"15min":0.45000000000000001},"min":{"1min":0.16700000000000001,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":0.81899999999999995,"5min":5.8899999999999997,"15min":6.383},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.315,"5min":0.28799999999999998,"15min":0.437},"min":{"1min":0.19700000000000001,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":6.0289999999999999},"last":0.33400000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:12:48 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.42599999999999999,"15min":0.499},"min":{"1min":0.252,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":0.52800000000000002,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.40600000000000003,"15min":0.624},"min":{"1min":0.23599999999999999,"5min":0.20000000000000001,"15min":0.19400000000000001},"max":{"1min":0.63300000000000001,"5min":2.1280000000000001,"15min":26.596},"last":0.374}]}]},{"osd":1,"up_from":8,"seq":34359739196,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4848916,"kb_used_data":3656844,"kb_used_omap":359,"kb_used_meta":1191704,"kb_avail":89522924,"statfs":{"total":96636764160,"available":91671474176,"internally_reserved":0,"allocated":3744608256,"data_stored":4290830110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1220304930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,6,7,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.373,"15min":0.40300000000000002},"min":{"1min":0.185,"5min":0.16800000000000001,"15min":0.16400000000000001},"max":{"1min":0.53200000000000003,"5min":3.1339999999999999,"15min":9.218},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.52800000000000002,"15min":0.51400000000000001},"min":{"1min":0.20499999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.5449999999999999,"5min":5.6369999999999996,"15min":5.6390000000000002},"last":0.23000000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.47399999999999998,"15min":0.54200000000000004},"min":{"1min":0.217,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.63800000000000001,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.40799999999999997,"15min":0.49299999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.193},"max":{"1min":0.56999999999999995,"5min":2.8029999999999999,"15min":19.521999999999998},"last":0.36399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3883433984,"data_stored":4455890944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3743240192,"data_stored":4290117651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7626924032,"data_stored":8746258451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:13:46.576 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:46.577 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:46.577 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:46.578 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:46.578 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:06.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:14:07.071 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:14:07.072 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:14:07.108 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2154,"stamp":"2023-10-21T05:14:05.663113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27613220072,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1891405,"num_read_kb":7387217,"num_write":1869955,"num_write_kb":7478799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76268,"ondisk_log_size":76268,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19707656,"kb_used_data":14955120,"kb_used_omap":1457,"kb_used_meta":4751054,"kb_avail":263407864,"statfs":{"total":289910292480,"available":269729652736,"internally_reserved":0,"allocated":15314042880,"data_stored":17551011168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":4865079492},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,7,8,12,19,26],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13328384,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3582,"num_read_kb":14224,"num_write":3631,"num_write_kb":14524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.263918"},"pg_stats":[{"pgid":"2.7","version":"81'9282","reported_seq":12812,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:13:59.268547+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:13:59.268547+0000","last_peered":"2023-10-21T05:13:59.268547+0000","last_clean":"2023-10-21T05:13:59.268547+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:13:59.268547+0000","last_undegraded":"2023-10-21T05:13:59.268547+0000","last_fullsized":"2023-10-21T05:13:59.268547+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":998,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:12:50.398154+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:12:50.398154+0000","last_peered":"2023-10-21T05:12:50.398154+0000","last_clean":"2023-10-21T05:12:50.398154+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:12:50.398154+0000","last_undegraded":"2023-10-21T05:12:50.398154+0000","last_fullsized":"2023-10-21T05:12:50.398154+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"81'15322","reported_seq":21040,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:13:59.269403+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:13:59.269403+0000","last_peered":"2023-10-21T05:13:59.269403+0000","last_clean":"2023-10-21T05:13:59.269403+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:13:59.269403+0000","last_undegraded":"2023-10-21T05:13:59.269403+0000","last_fullsized":"2023-10-21T05:13:59.269403+0000","mapping_epoch":12,"log_start":"75'5300","ondisk_log_start":"75'5300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"81'3174","reported_seq":4946,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:13:55.401789+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:13:55.401789+0000","last_peered":"2023-10-21T05:13:55.401789+0000","last_clean":"2023-10-21T05:13:55.401789+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:13:55.401789+0000","last_undegraded":"2023-10-21T05:13:55.401789+0000","last_fullsized":"2023-10-21T05:13:55.401789+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3174,"log_dups_size":0,"ondisk_log_size":3174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"82'472238","reported_seq":942213,"reported_epoch":82,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:03.084231+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:14:03.084231+0000","last_peered":"2023-10-21T05:14:03.084231+0000","last_clean":"2023-10-21T05:14:03.084231+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:14:03.084231+0000","last_undegraded":"2023-10-21T05:14:03.084231+0000","last_fullsized":"2023-10-21T05:14:03.084231+0000","mapping_epoch":15,"log_start":"80'462215","ondisk_log_start":"80'462215","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1522,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 176s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6940422144,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":471463,"num_read_kb":1855748,"num_write":469475,"num_write_kb":1877900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"82'15611","reported_seq":21549,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:13:59.997553+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:13:59.997553+0000","last_peered":"2023-10-21T05:13:59.997553+0000","last_clean":"2023-10-21T05:13:59.997553+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:13:59.997553+0000","last_undegraded":"2023-10-21T05:13:59.997553+0000","last_fullsized":"2023-10-21T05:13:59.997553+0000","mapping_epoch":12,"log_start":"75'5600","ondisk_log_start":"75'5600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"82'466345","reported_seq":929454,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:02.621555+0000","last_change":"2023-10-21T05:14:02.491297+0000","last_active":"2023-10-21T05:14:02.621555+0000","last_peered":"2023-10-21T05:14:02.621555+0000","last_clean":"2023-10-21T05:14:02.621555+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:14:02.621555+0000","last_undegraded":"2023-10-21T05:14:02.621555+0000","last_fullsized":"2023-10-21T05:14:02.621555+0000","mapping_epoch":15,"log_start":"80'456316","ondisk_log_start":"80'456316","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6845956096,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":464620,"num_read_kb":1827988,"num_write":464590,"num_write_kb":1858360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6196,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:13:59.269000+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:13:59.269000+0000","last_peered":"2023-10-21T05:13:59.269000+0000","last_clean":"2023-10-21T05:13:59.269000+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:13:59.269000+0000","last_undegraded":"2023-10-21T05:13:59.269000+0000","last_fullsized":"2023-10-21T05:13:59.269000+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"82'485106","reported_seq":966337,"reported_epoch":82,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:03.066637+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:14:03.066637+0000","last_peered":"2023-10-21T05:14:03.066637+0000","last_clean":"2023-10-21T05:14:03.066637+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:14:03.066637+0000","last_undegraded":"2023-10-21T05:14:03.066637+0000","last_fullsized":"2023-10-21T05:14:03.066637+0000","mapping_epoch":15,"log_start":"80'475025","ondisk_log_start":"80'475025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1508,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 177s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7135567872,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":482775,"num_read_kb":1900200,"num_write":482272,"num_write_kb":1929088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":157,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:27.801952+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:13:27.801952+0000","last_peered":"2023-10-21T05:13:27.801952+0000","last_clean":"2023-10-21T05:13:27.801952+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:13:27.801952+0000","last_undegraded":"2023-10-21T05:13:27.801952+0000","last_fullsized":"2023-10-21T05:13:27.801952+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"82'455465","reported_seq":908428,"reported_epoch":82,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:02.626882+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:14:02.626882+0000","last_peered":"2023-10-21T05:14:02.626882+0000","last_clean":"2023-10-21T05:14:02.626882+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:14:02.626882+0000","last_undegraded":"2023-10-21T05:14:02.626882+0000","last_fullsized":"2023-10-21T05:14:02.626882+0000","mapping_epoch":15,"log_start":"80'445417","ondisk_log_start":"80'445417","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1420,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 172s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6690680851,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":454382,"num_read_kb":1787804,"num_write":452812,"num_write_kb":1811242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"81'3152","reported_seq":4459,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:13:55.401725+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:13:55.401725+0000","last_peered":"2023-10-21T05:13:55.401725+0000","last_clean":"2023-10-21T05:13:55.401725+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:13:55.401725+0000","last_undegraded":"2023-10-21T05:13:55.401725+0000","last_fullsized":"2023-10-21T05:13:55.401725+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3152,"log_dups_size":0,"ondisk_log_size":3152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":254,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:22.483569+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:13:22.483569+0000","last_peered":"2023-10-21T05:13:22.483569+0000","last_clean":"2023-10-21T05:13:22.483569+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:13:22.483569+0000","last_undegraded":"2023-10-21T05:13:22.483569+0000","last_fullsized":"2023-10-21T05:13:22.483569+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27612626963,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1873240,"num_read_kb":7371740,"num_write":1869149,"num_write_kb":7476590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15312195584,"data_stored":17550864422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40181,"ondisk_log_size":40181,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":35988,"ondisk_log_size":35988,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739202,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5183600,"kb_used_data":3807848,"kb_used_omap":373,"kb_used_meta":1375370,"kb_avail":89188240,"statfs":{"total":96636764160,"available":91328757760,"internally_reserved":0,"allocated":3899236352,"data_stored":4471118512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1408379440},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,8,8,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35299999999999998,"15min":0.39600000000000002},"min":{"1min":0.16800000000000001,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":0.81200000000000006,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.49299999999999999,"15min":0.435},"min":{"1min":0.19600000000000001,"5min":0.191,"15min":0.187},"max":{"1min":6.2990000000000004,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.28499999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.53000000000000003,"15min":0.56399999999999995},"min":{"1min":0.214,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":0.54400000000000004,"5min":5.4100000000000001,"15min":14.318},"last":0.318},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.57299999999999995,"15min":0.62},"min":{"1min":0.28599999999999998,"5min":0.25,"15min":0.20899999999999999},"max":{"1min":0.58799999999999997,"5min":5.4450000000000003,"15min":14.343},"last":0.34499999999999997}]}]},{"osd":2,"up_from":8,"seq":34359739201,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9606916,"kb_used_data":7476348,"kb_used_omap":724,"kb_used_meta":2129835,"kb_avail":84764924,"statfs":{"total":96636764160,"available":86799282176,"internally_reserved":0,"allocated":7655780352,"data_stored":8774648984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":2180951666},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:53 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.45400000000000001,"15min":0.45700000000000002},"min":{"1min":0.17899999999999999,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":0.77000000000000002,"5min":5.8899999999999997,"15min":6.383},"last":4.3029999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.30399999999999999,"15min":0.41999999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":0.59499999999999997,"5min":0.60799999999999998,"15min":6.0289999999999999},"last":0.34499999999999997}]},{"osd":1,"last update":"Sat Oct 21 05:13:53 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.46200000000000002,"15min":0.51100000000000001},"min":{"1min":0.27700000000000002,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":2.9020000000000001,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.41899999999999998,"15min":0.60699999999999998},"min":{"1min":0.28899999999999998,"5min":0.21099999999999999,"15min":0.19400000000000001},"max":{"1min":2.8580000000000001,"5min":2.8580000000000001,"15min":26.596},"last":0.309}]}]},{"osd":1,"up_from":8,"seq":34359739200,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4917140,"kb_used_data":3670924,"kb_used_omap":359,"kb_used_meta":1245848,"kb_avail":89454700,"statfs":{"total":96636764160,"available":91601612800,"internally_reserved":0,"allocated":3759026176,"data_stored":4305243672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1275748386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,2,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.373,"15min":0.40300000000000002},"min":{"1min":0.185,"5min":0.16800000000000001,"15min":0.16400000000000001},"max":{"1min":0.53200000000000003,"5min":3.1339999999999999,"15min":9.218},"last":0.624},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.52800000000000002,"15min":0.51400000000000001},"min":{"1min":0.20499999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.5449999999999999,"5min":5.6369999999999996,"15min":5.6390000000000002},"last":0.92600000000000005}]},{"osd":2,"last update":"Sat Oct 21 05:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.47399999999999998,"15min":0.54200000000000004},"min":{"1min":0.217,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.63800000000000001,"5min":6.4119999999999999,"15min":19.559999999999999},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.40799999999999997,"15min":0.49299999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.193},"max":{"1min":0.56999999999999995,"5min":2.8029999999999999,"15min":19.521999999999998},"last":0.42199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3898355712,"data_stored":4470812672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3757686784,"data_stored":4304564243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7656153088,"data_stored":8775487507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:14:07.110 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:07.111 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:07.111 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:07.111 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:07.112 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:13.400 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.395+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:14:13.403 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.399+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:14:13.404 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.399+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:14:13.404 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.399+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:14:13.410 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.403+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:14:13.886 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.879+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:14:13.887 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.883+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:14:13.901 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.895+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:14:13.902 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.895+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:14:13.902 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.895+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:14:13.902 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.895+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:14:13.903 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.895+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:14:13.903 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.895+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:14:13.903 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.895+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:14:13.904 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.895+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:14:13.904 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:14:13.895+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:14:27.113 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:14:27.631 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:14:27.632 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:14:27.674 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2164,"stamp":"2023-10-21T05:14:25.673027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27639631080,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1898622,"num_read_kb":7415909,"num_write":1877198,"num_write_kb":7507771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76512,"ondisk_log_size":76512,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19863852,"kb_used_data":15012820,"kb_used_omap":1457,"kb_used_meta":4849550,"kb_avail":263251668,"statfs":{"total":289910292480,"available":269569708032,"internally_reserved":0,"allocated":15373127680,"data_stored":17610095968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1492796,"internal_metadata":4965939396},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,3,6,11,7,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13008896,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3544,"num_read_kb":14068,"num_write":3579,"num_write_kb":14316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008685"},"pg_stats":[{"pgid":"2.7","version":"82'9321","reported_seq":12863,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:19.751732+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:14:19.751732+0000","last_peered":"2023-10-21T05:14:19.751732+0000","last_clean":"2023-10-21T05:14:19.751732+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:14:19.751732+0000","last_undegraded":"2023-10-21T05:14:19.751732+0000","last_fullsized":"2023-10-21T05:14:19.751732+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":998,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:12:50.398154+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:12:50.398154+0000","last_peered":"2023-10-21T05:12:50.398154+0000","last_clean":"2023-10-21T05:12:50.398154+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:12:50.398154+0000","last_undegraded":"2023-10-21T05:12:50.398154+0000","last_fullsized":"2023-10-21T05:12:50.398154+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"82'15385","reported_seq":21120,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:19.752252+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:14:19.752252+0000","last_peered":"2023-10-21T05:14:19.752252+0000","last_clean":"2023-10-21T05:14:19.752252+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:14:19.752252+0000","last_undegraded":"2023-10-21T05:14:19.752252+0000","last_fullsized":"2023-10-21T05:14:19.752252+0000","mapping_epoch":12,"log_start":"75'5300","ondisk_log_start":"75'5300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"82'3188","reported_seq":4965,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:17.635054+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:14:17.635054+0000","last_peered":"2023-10-21T05:14:17.635054+0000","last_clean":"2023-10-21T05:14:17.635054+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:14:17.635054+0000","last_undegraded":"2023-10-21T05:14:17.635054+0000","last_fullsized":"2023-10-21T05:14:17.635054+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3188,"log_dups_size":0,"ondisk_log_size":3188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"82'474107","reported_seq":945989,"reported_epoch":82,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:23.163873+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:14:23.163873+0000","last_peered":"2023-10-21T05:14:23.163873+0000","last_clean":"2023-10-21T05:14:23.163873+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:14:23.163873+0000","last_undegraded":"2023-10-21T05:14:23.163873+0000","last_fullsized":"2023-10-21T05:14:23.163873+0000","mapping_epoch":15,"log_start":"80'464015","ondisk_log_start":"80'464015","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1656,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 196s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6946877440,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":473345,"num_read_kb":1863228,"num_write":471258,"num_write_kb":1885032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"82'15671","reported_seq":21630,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:21.937503+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:14:21.937503+0000","last_peered":"2023-10-21T05:14:21.937503+0000","last_clean":"2023-10-21T05:14:21.937503+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:14:21.937503+0000","last_undegraded":"2023-10-21T05:14:21.937503+0000","last_fullsized":"2023-10-21T05:14:21.937503+0000","mapping_epoch":12,"log_start":"75'5600","ondisk_log_start":"75'5600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"82'468212","reported_seq":933102,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:22.716835+0000","last_change":"2023-10-21T05:14:22.502894+0000","last_active":"2023-10-21T05:14:22.716835+0000","last_peered":"2023-10-21T05:14:22.716835+0000","last_clean":"2023-10-21T05:14:22.716835+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:14:22.716835+0000","last_undegraded":"2023-10-21T05:14:22.716835+0000","last_fullsized":"2023-10-21T05:14:22.716835+0000","mapping_epoch":15,"log_start":"80'458116","ondisk_log_start":"80'458116","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6852788224,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":466399,"num_read_kb":1835048,"num_write":466457,"num_write_kb":1865828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6216,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:19.752287+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:14:19.752287+0000","last_peered":"2023-10-21T05:14:19.752287+0000","last_clean":"2023-10-21T05:14:19.752287+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:14:19.752287+0000","last_undegraded":"2023-10-21T05:14:19.752287+0000","last_fullsized":"2023-10-21T05:14:19.752287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"82'487061","reported_seq":970170,"reported_epoch":82,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:23.163556+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:14:23.163556+0000","last_peered":"2023-10-21T05:14:23.163556+0000","last_clean":"2023-10-21T05:14:23.163556+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:14:23.163556+0000","last_undegraded":"2023-10-21T05:14:23.163556+0000","last_fullsized":"2023-10-21T05:14:23.163556+0000","mapping_epoch":15,"log_start":"80'477025","ondisk_log_start":"80'477025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1649,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 197s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7142457344,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":484640,"num_read_kb":1907628,"num_write":484140,"num_write_kb":1936560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":157,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:27.801952+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:13:27.801952+0000","last_peered":"2023-10-21T05:13:27.801952+0000","last_clean":"2023-10-21T05:13:27.801952+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:13:27.801952+0000","last_undegraded":"2023-10-21T05:13:27.801952+0000","last_fullsized":"2023-10-21T05:13:27.801952+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"82'457328","reported_seq":911998,"reported_epoch":82,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:22.699427+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:14:22.699427+0000","last_peered":"2023-10-21T05:14:22.699427+0000","last_clean":"2023-10-21T05:14:22.699427+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:14:22.699427+0000","last_undegraded":"2023-10-21T05:14:22.699427+0000","last_fullsized":"2023-10-21T05:14:22.699427+0000","mapping_epoch":15,"log_start":"80'447317","ondisk_log_start":"80'447317","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1646,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 192s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6696914963,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":456073,"num_read_kb":1794528,"num_write":454537,"num_write_kb":1818142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"82'3166","reported_seq":4478,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:17.647736+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:14:17.647736+0000","last_peered":"2023-10-21T05:14:17.647736+0000","last_clean":"2023-10-21T05:14:17.647736+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:14:17.647736+0000","last_undegraded":"2023-10-21T05:14:17.647736+0000","last_fullsized":"2023-10-21T05:14:17.647736+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3166,"log_dups_size":0,"ondisk_log_size":3166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":254,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:22.483569+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:13:22.483569+0000","last_peered":"2023-10-21T05:13:22.483569+0000","last_clean":"2023-10-21T05:13:22.483569+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:13:22.483569+0000","last_undegraded":"2023-10-21T05:13:22.483569+0000","last_fullsized":"2023-10-21T05:13:22.483569+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27639037971,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1880457,"num_read_kb":7400432,"num_write":1876392,"num_write_kb":7505562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15371636736,"data_stored":17610305574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40235,"ondisk_log_size":40235,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36178,"ondisk_log_size":36178,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739206,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5212096,"kb_used_data":3822456,"kb_used_omap":373,"kb_used_meta":1389258,"kb_avail":89159744,"statfs":{"total":96636764160,"available":91299577856,"internally_reserved":0,"allocated":3914194944,"data_stored":4486077104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1422600752},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,9,7,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35299999999999998,"15min":0.39600000000000002},"min":{"1min":0.16800000000000001,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":0.81200000000000006,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.49299999999999999,"15min":0.435},"min":{"1min":0.19600000000000001,"5min":0.191,"15min":0.187},"max":{"1min":6.2990000000000004,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.35299999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.53000000000000003,"15min":0.56399999999999995},"min":{"1min":0.214,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":0.54400000000000004,"5min":5.4100000000000001,"15min":14.318},"last":0.214},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.57299999999999995,"15min":0.62},"min":{"1min":0.28599999999999998,"5min":0.25,"15min":0.20899999999999999},"max":{"1min":0.58799999999999997,"5min":5.4450000000000003,"15min":14.343},"last":0.27800000000000002}]}]},{"osd":2,"up_from":8,"seq":34359739205,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9702188,"kb_used_data":7505060,"kb_used_omap":724,"kb_used_meta":2196395,"kb_avail":84669652,"statfs":{"total":96636764160,"available":86701723648,"internally_reserved":0,"allocated":7685181440,"data_stored":8804050072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":741774,"internal_metadata":2249109106},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:53 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.45400000000000001,"15min":0.45700000000000002},"min":{"1min":0.17899999999999999,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":0.77000000000000002,"5min":5.8899999999999997,"15min":6.383},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.30399999999999999,"15min":0.41999999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":0.59499999999999997,"5min":0.60799999999999998,"15min":6.0289999999999999},"last":0.32800000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:13:53 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.46200000000000002,"15min":0.51100000000000001},"min":{"1min":0.27700000000000002,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":2.9020000000000001,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.374},{"interface":"front","average":{"1min":0.502,"5min":0.41899999999999998,"15min":0.60699999999999998},"min":{"1min":0.28899999999999998,"5min":0.21099999999999999,"15min":0.19400000000000001},"max":{"1min":2.8580000000000001,"5min":2.8580000000000001,"15min":26.596},"last":0.29599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739204,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4949568,"kb_used_data":3685304,"kb_used_omap":359,"kb_used_meta":1263896,"kb_avail":89422272,"statfs":{"total":96636764160,"available":91568406528,"internally_reserved":0,"allocated":3773751296,"data_stored":4319968792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1294229538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.373,"15min":0.40300000000000002},"min":{"1min":0.185,"5min":0.16800000000000001,"15min":0.16400000000000001},"max":{"1min":0.53200000000000003,"5min":3.1339999999999999,"15min":9.218},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.52800000000000002,"15min":0.51400000000000001},"min":{"1min":0.20499999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.5449999999999999,"5min":5.6369999999999996,"15min":5.6390000000000002},"last":0.29299999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.47399999999999998,"15min":0.54200000000000004},"min":{"1min":0.217,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.63800000000000001,"5min":6.4119999999999999,"15min":19.559999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.40799999999999997,"15min":0.49299999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.193},"max":{"1min":0.56999999999999995,"5min":2.8029999999999999,"15min":19.521999999999998},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3913310208,"data_stored":4485767168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3772391424,"data_stored":4319268883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7685935104,"data_stored":8805269523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:14:27.676 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:27.676 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:27.676 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:27.677 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:27.677 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:47.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:14:48.195 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:14:48.196 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:14:48.224 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2175,"stamp":"2023-10-21T05:14:47.676719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27666193640,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1905563,"num_read_kb":7443541,"num_write":1884465,"num_write_kb":7536839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76434,"ondisk_log_size":76434,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19054808,"kb_used_data":15071616,"kb_used_omap":1459,"kb_used_meta":3981708,"kb_avail":264060712,"statfs":{"total":289910292480,"available":270398169088,"internally_reserved":0,"allocated":15433334784,"data_stored":17670303072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":4077269556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,5,6,13,29,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13467648,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3519,"num_read_kb":14012,"num_write":3670,"num_write_kb":14680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001953"},"pg_stats":[{"pgid":"2.7","version":"82'9351","reported_seq":12906,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:41.159866+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:14:41.159866+0000","last_peered":"2023-10-21T05:14:41.159866+0000","last_clean":"2023-10-21T05:14:41.159866+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:14:41.159866+0000","last_undegraded":"2023-10-21T05:14:41.159866+0000","last_fullsized":"2023-10-21T05:14:41.159866+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9351,"log_dups_size":0,"ondisk_log_size":9351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":999,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:13:55.401767+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:13:55.401767+0000","last_peered":"2023-10-21T05:13:55.401767+0000","last_clean":"2023-10-21T05:13:55.401767+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:13:55.401767+0000","last_undegraded":"2023-10-21T05:13:55.401767+0000","last_fullsized":"2023-10-21T05:13:55.401767+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"82'15435","reported_seq":21190,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:41.165732+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:14:41.165732+0000","last_peered":"2023-10-21T05:14:41.165732+0000","last_clean":"2023-10-21T05:14:41.165732+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:14:41.165732+0000","last_undegraded":"2023-10-21T05:14:41.165732+0000","last_fullsized":"2023-10-21T05:14:41.165732+0000","mapping_epoch":12,"log_start":"75'5400","ondisk_log_start":"75'5400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"82'3198","reported_seq":4979,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:40.227590+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:14:40.227590+0000","last_peered":"2023-10-21T05:14:40.227590+0000","last_clean":"2023-10-21T05:14:40.227590+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:14:40.227590+0000","last_undegraded":"2023-10-21T05:14:40.227590+0000","last_fullsized":"2023-10-21T05:14:40.227590+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3198,"log_dups_size":0,"ondisk_log_size":3198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"82'476010","reported_seq":949670,"reported_epoch":82,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:43.296960+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:14:43.296960+0000","last_peered":"2023-10-21T05:14:43.296960+0000","last_clean":"2023-10-21T05:14:43.296960+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:14:43.296960+0000","last_undegraded":"2023-10-21T05:14:43.296960+0000","last_fullsized":"2023-10-21T05:14:43.296960+0000","mapping_epoch":15,"log_start":"81'465916","ondisk_log_start":"81'465916","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1831,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 216s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6953377792,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":475104,"num_read_kb":1870228,"num_write":473052,"num_write_kb":1892208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"82'15731","reported_seq":21710,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:42.435008+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:14:42.435008+0000","last_peered":"2023-10-21T05:14:42.435008+0000","last_clean":"2023-10-21T05:14:42.435008+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:14:42.435008+0000","last_undegraded":"2023-10-21T05:14:42.435008+0000","last_fullsized":"2023-10-21T05:14:42.435008+0000","mapping_epoch":12,"log_start":"75'5700","ondisk_log_start":"75'5700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"82'470033","reported_seq":936629,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:42.757635+0000","last_change":"2023-10-21T05:14:36.780867+0000","last_active":"2023-10-21T05:14:42.757635+0000","last_peered":"2023-10-21T05:14:42.757635+0000","last_clean":"2023-10-21T05:14:42.757635+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:14:42.757635+0000","last_undegraded":"2023-10-21T05:14:42.757635+0000","last_fullsized":"2023-10-21T05:14:42.757635+0000","mapping_epoch":15,"log_start":"81'460016","ondisk_log_start":"81'460016","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6859407360,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":468098,"num_read_kb":1841820,"num_write":468278,"num_write_kb":1873112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6236,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:40.258181+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:14:40.258181+0000","last_peered":"2023-10-21T05:14:40.258181+0000","last_clean":"2023-10-21T05:14:40.258181+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:14:40.258181+0000","last_undegraded":"2023-10-21T05:14:40.258181+0000","last_fullsized":"2023-10-21T05:14:40.258181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"82'489058","reported_seq":973991,"reported_epoch":82,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:43.278089+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:14:43.278089+0000","last_peered":"2023-10-21T05:14:43.278089+0000","last_clean":"2023-10-21T05:14:43.278089+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:14:43.278089+0000","last_undegraded":"2023-10-21T05:14:43.278089+0000","last_fullsized":"2023-10-21T05:14:43.278089+0000","mapping_epoch":15,"log_start":"81'479025","ondisk_log_start":"81'479025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1784,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 217s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7149477888,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":486451,"num_read_kb":1914836,"num_write":486053,"num_write_kb":1944212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":157,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:27.801952+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:13:27.801952+0000","last_peered":"2023-10-21T05:13:27.801952+0000","last_clean":"2023-10-21T05:13:27.801952+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:13:27.801952+0000","last_undegraded":"2023-10-21T05:13:27.801952+0000","last_fullsized":"2023-10-21T05:13:27.801952+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"82'459170","reported_seq":915528,"reported_epoch":82,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:42.788430+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:14:42.788430+0000","last_peered":"2023-10-21T05:14:42.788430+0000","last_clean":"2023-10-21T05:14:42.788430+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:14:42.788430+0000","last_undegraded":"2023-10-21T05:14:42.788430+0000","last_fullsized":"2023-10-21T05:14:42.788430+0000","mapping_epoch":15,"log_start":"81'449117","ondisk_log_start":"81'449117","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1806,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 213s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6703337491,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":457745,"num_read_kb":1801180,"num_write":456276,"num_write_kb":1825098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"82'3176","reported_seq":4492,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:40.227435+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:14:40.227435+0000","last_peered":"2023-10-21T05:14:40.227435+0000","last_clean":"2023-10-21T05:14:40.227435+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:14:40.227435+0000","last_undegraded":"2023-10-21T05:14:40.227435+0000","last_fullsized":"2023-10-21T05:14:40.227435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3176,"log_dups_size":0,"ondisk_log_size":3176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":254,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:22.483569+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:13:22.483569+0000","last_peered":"2023-10-21T05:13:22.483569+0000","last_clean":"2023-10-21T05:13:22.483569+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:13:22.483569+0000","last_undegraded":"2023-10-21T05:13:22.483569+0000","last_fullsized":"2023-10-21T05:13:22.483569+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27665600531,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1887398,"num_read_kb":7428064,"num_write":1883659,"num_write_kb":7534630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15431217152,"data_stored":17669885990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40197,"ondisk_log_size":40197,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36138,"ondisk_log_size":36138,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739210,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5287584,"kb_used_data":3837272,"kb_used_omap":373,"kb_used_meta":1449930,"kb_avail":89084256,"statfs":{"total":96636764160,"available":91222278144,"internally_reserved":0,"allocated":3929366528,"data_stored":4501248688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1484728880},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,9,10,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35299999999999998,"15min":0.39600000000000002},"min":{"1min":0.16800000000000001,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":0.81200000000000006,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.49299999999999999,"15min":0.435},"min":{"1min":0.19600000000000001,"5min":0.191,"15min":0.187},"max":{"1min":6.2990000000000004,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.53000000000000003,"15min":0.56399999999999995},"min":{"1min":0.214,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":0.54400000000000004,"5min":5.4100000000000001,"15min":14.318},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.57299999999999995,"15min":0.62},"min":{"1min":0.28599999999999998,"5min":0.25,"15min":0.20899999999999999},"max":{"1min":0.58799999999999997,"5min":5.4450000000000003,"15min":14.343},"last":0.35699999999999998}]}]},{"osd":2,"up_from":8,"seq":34359739209,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8789968,"kb_used_data":7535240,"kb_used_omap":726,"kb_used_meta":1253993,"kb_avail":85581872,"statfs":{"total":96636764160,"available":87635836928,"internally_reserved":0,"allocated":7716085760,"data_stored":8834954392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":1284089826},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:53 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.45400000000000001,"15min":0.45700000000000002},"min":{"1min":0.17899999999999999,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":0.77000000000000002,"5min":5.8899999999999997,"15min":6.383},"last":0.249},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.30399999999999999,"15min":0.41999999999999998},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":0.59499999999999997,"5min":0.60799999999999998,"15min":6.0289999999999999},"last":0.30599999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:13:53 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.46200000000000002,"15min":0.51100000000000001},"min":{"1min":0.27700000000000002,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":2.9020000000000001,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.41899999999999998,"15min":0.60699999999999998},"min":{"1min":0.28899999999999998,"5min":0.21099999999999999,"15min":0.19400000000000001},"max":{"1min":2.8580000000000001,"5min":2.8580000000000001,"15min":26.596},"last":0.35199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739208,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4977256,"kb_used_data":3699104,"kb_used_omap":359,"kb_used_meta":1277784,"kb_avail":89394584,"statfs":{"total":96636764160,"available":91540054016,"internally_reserved":0,"allocated":3787882496,"data_stored":4334099992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1308450850},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,0,2,19],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34300000000000003,"15min":0.40799999999999997},"min":{"1min":0.23699999999999999,"5min":0.16800000000000001,"15min":0.16400000000000001},"max":{"1min":0.624,"5min":3.1339999999999999,"15min":9.218},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.49199999999999999,"15min":0.51600000000000001},"min":{"1min":0.186,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.92600000000000005,"5min":5.6369999999999996,"15min":5.6390000000000002},"last":0.22700000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:14:26 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40000000000000002,"15min":0.53400000000000003},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.0289999999999999,"5min":1.782,"15min":19.559999999999999},"last":0.215},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.38400000000000001,"15min":0.49399999999999999},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.19800000000000001},"max":{"1min":0.55800000000000005,"5min":2.8029999999999999,"15min":19.521999999999998},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3928510464,"data_stored":4500967424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3786969088,"data_stored":4333846547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7715737600,"data_stored":8835072019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:14:48.226 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:48.227 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:48.227 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:48.227 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:48.228 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:15:05.407 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:05.406+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:15:05.407 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:05.406+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:15:08.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:15:08.688 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:15:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:15:08.722 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2186,"stamp":"2023-10-21T05:15:07.679986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27692428520,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1912189,"num_read_kb":7469913,"num_write":1891628,"num_write_kb":7565491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76594,"ondisk_log_size":76594,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19270432,"kb_used_data":15128968,"kb_used_omap":1459,"kb_used_meta":4139980,"kb_avail":263845088,"statfs":{"total":289910292480,"available":270177370112,"internally_reserved":0,"allocated":15492063232,"data_stored":17729031520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":4239340084},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,8,12,25,12,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12955648,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3238,"num_read_kb":12892,"num_write":3554,"num_write_kb":14216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001646"},"pg_stats":[{"pgid":"2.7","version":"82'9369","reported_seq":12936,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:15:00.707750+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:15:00.707750+0000","last_peered":"2023-10-21T05:15:00.707750+0000","last_clean":"2023-10-21T05:15:00.707750+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:15:00.707750+0000","last_undegraded":"2023-10-21T05:15:00.707750+0000","last_fullsized":"2023-10-21T05:15:00.707750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9369,"log_dups_size":0,"ondisk_log_size":9369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":999,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:13:55.401767+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:13:55.401767+0000","last_peered":"2023-10-21T05:13:55.401767+0000","last_clean":"2023-10-21T05:13:55.401767+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:13:55.401767+0000","last_undegraded":"2023-10-21T05:13:55.401767+0000","last_fullsized":"2023-10-21T05:13:55.401767+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"82'15465","reported_seq":21240,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:15:00.708143+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:15:00.708143+0000","last_peered":"2023-10-21T05:15:00.708143+0000","last_clean":"2023-10-21T05:15:00.708143+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:15:00.708143+0000","last_undegraded":"2023-10-21T05:15:00.708143+0000","last_fullsized":"2023-10-21T05:15:00.708143+0000","mapping_epoch":12,"log_start":"75'5400","ondisk_log_start":"75'5400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"82'3204","reported_seq":4989,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:15:00.197452+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:15:00.197452+0000","last_peered":"2023-10-21T05:15:00.197452+0000","last_clean":"2023-10-21T05:15:00.197452+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:15:00.197452+0000","last_undegraded":"2023-10-21T05:15:00.197452+0000","last_fullsized":"2023-10-21T05:15:00.197452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3204,"log_dups_size":0,"ondisk_log_size":3204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"82'477911","reported_seq":953257,"reported_epoch":82,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:03.442839+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:15:03.442839+0000","last_peered":"2023-10-21T05:15:03.442839+0000","last_clean":"2023-10-21T05:15:03.442839+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:15:03.442839+0000","last_undegraded":"2023-10-21T05:15:03.442839+0000","last_fullsized":"2023-10-21T05:15:03.442839+0000","mapping_epoch":15,"log_start":"81'467816","ondisk_log_start":"81'467816","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2046,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 236s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6959931392,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":476774,"num_read_kb":1876876,"num_write":474816,"num_write_kb":1899264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"82'15797","reported_seq":21796,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:15:01.594758+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:15:01.594758+0000","last_peered":"2023-10-21T05:15:01.594758+0000","last_clean":"2023-10-21T05:15:01.594758+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:15:01.594758+0000","last_undegraded":"2023-10-21T05:15:01.594758+0000","last_fullsized":"2023-10-21T05:15:01.594758+0000","mapping_epoch":12,"log_start":"75'5700","ondisk_log_start":"75'5700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"82'471840","reported_seq":940080,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:15:02.889931+0000","last_change":"2023-10-21T05:15:02.058404+0000","last_active":"2023-10-21T05:15:02.889931+0000","last_peered":"2023-10-21T05:15:02.889931+0000","last_clean":"2023-10-21T05:15:02.889931+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:15:02.889931+0000","last_undegraded":"2023-10-21T05:15:02.889931+0000","last_fullsized":"2023-10-21T05:15:02.889931+0000","mapping_epoch":15,"log_start":"81'461816","ondisk_log_start":"81'461816","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6865965056,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":469736,"num_read_kb":1848344,"num_write":470085,"num_write_kb":1880340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6256,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:15:00.708740+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:15:00.708740+0000","last_peered":"2023-10-21T05:15:00.708740+0000","last_clean":"2023-10-21T05:15:00.708740+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:15:00.708740+0000","last_undegraded":"2023-10-21T05:15:00.708740+0000","last_fullsized":"2023-10-21T05:15:00.708740+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"82'491060","reported_seq":977728,"reported_epoch":82,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:03.440724+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:15:03.440724+0000","last_peered":"2023-10-21T05:15:03.440724+0000","last_clean":"2023-10-21T05:15:03.440724+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:15:03.440724+0000","last_undegraded":"2023-10-21T05:15:03.440724+0000","last_fullsized":"2023-10-21T05:15:03.440724+0000","mapping_epoch":15,"log_start":"81'481025","ondisk_log_start":"81'481025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1989,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 237s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7156412416,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":488170,"num_read_kb":1921676,"num_write":487928,"num_write_kb":1951712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":157,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:27.801952+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:13:27.801952+0000","last_peered":"2023-10-21T05:13:27.801952+0000","last_clean":"2023-10-21T05:13:27.801952+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:13:27.801952+0000","last_undegraded":"2023-10-21T05:13:27.801952+0000","last_fullsized":"2023-10-21T05:13:27.801952+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"82'460995","reported_seq":918970,"reported_epoch":82,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:02.903434+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:15:02.903434+0000","last_peered":"2023-10-21T05:15:02.903434+0000","last_clean":"2023-10-21T05:15:02.903434+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:15:02.903434+0000","last_undegraded":"2023-10-21T05:15:02.903434+0000","last_fullsized":"2023-10-21T05:15:02.903434+0000","mapping_epoch":15,"log_start":"81'450918","ondisk_log_start":"81'450918","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1976,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 233s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6709526547,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":459344,"num_read_kb":1807540,"num_write":457993,"num_write_kb":1831966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"82'3182","reported_seq":4502,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:15:00.190183+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:15:00.190183+0000","last_peered":"2023-10-21T05:15:00.190183+0000","last_clean":"2023-10-21T05:15:00.190183+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:15:00.190183+0000","last_undegraded":"2023-10-21T05:15:00.190183+0000","last_fullsized":"2023-10-21T05:15:00.190183+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3182,"log_dups_size":0,"ondisk_log_size":3182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":254,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-21T05:13:22.483569+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:13:22.483569+0000","last_peered":"2023-10-21T05:13:22.483569+0000","last_clean":"2023-10-21T05:13:22.483569+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:13:22.483569+0000","last_undegraded":"2023-10-21T05:13:22.483569+0000","last_fullsized":"2023-10-21T05:13:22.483569+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27691835411,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1894024,"num_read_kb":7454436,"num_write":1890822,"num_write_kb":7563282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15489888256,"data_stored":17728557094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40231,"ondisk_log_size":40231,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36264,"ondisk_log_size":36264,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739214,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5338152,"kb_used_data":3851744,"kb_used_omap":373,"kb_used_meta":1486026,"kb_avail":89033688,"statfs":{"total":96636764160,"available":91170496512,"internally_reserved":0,"allocated":3944185856,"data_stored":4516068016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1521691184},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,3,11,12,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:14:59 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.36499999999999999,"15min":0.40100000000000002},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":1.675,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.251},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.46400000000000002,"15min":0.44400000000000001},"min":{"1min":0.19500000000000001,"5min":0.191,"15min":0.187},"max":{"1min":2.02,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.83599999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:14:59 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.39600000000000002,"15min":0.53600000000000003},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":0.59999999999999998,"5min":2.488,"15min":14.318},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.439,"15min":0.59299999999999997},"min":{"1min":0.26900000000000002,"5min":0.25,"15min":0.20899999999999999},"max":{"1min":0.622,"5min":2.5209999999999999,"15min":14.343},"last":0.35499999999999998}]}]},{"osd":2,"up_from":8,"seq":34359739213,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8886716,"kb_used_data":7563956,"kb_used_omap":726,"kb_used_meta":1322025,"kb_avail":85485124,"statfs":{"total":96636764160,"available":87536766976,"internally_reserved":0,"allocated":7745490944,"data_stored":8864359576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":1353754594},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.433,"15min":0.45200000000000001},"min":{"1min":0.16600000000000001,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":4.3029999999999999,"5min":4.3029999999999999,"15min":5.8899999999999997},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.313,"15min":0.36899999999999999},"min":{"1min":0.20599999999999999,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":0.56299999999999994,"5min":0.60799999999999998,"15min":4.9029999999999996},"last":0.23100000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:14:56 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.437,"15min":0.48499999999999999},"min":{"1min":0.23499999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":1.8009999999999999,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.40400000000000003,"15min":0.57199999999999995},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.19400000000000001},"max":{"1min":1.766,"5min":2.8580000000000001,"15min":26.596},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739212,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5045564,"kb_used_data":3713268,"kb_used_omap":359,"kb_used_meta":1331928,"kb_avail":89326276,"statfs":{"total":96636764160,"available":91470106624,"internally_reserved":0,"allocated":3802386432,"data_stored":4348603928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1363894306},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,7,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34300000000000003,"15min":0.40799999999999997},"min":{"1min":0.23699999999999999,"5min":0.16800000000000001,"15min":0.16400000000000001},"max":{"1min":0.624,"5min":3.1339999999999999,"15min":9.218},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.49199999999999999,"15min":0.51600000000000001},"min":{"1min":0.186,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.92600000000000005,"5min":5.6369999999999996,"15min":5.6390000000000002},"last":0.22}]},{"osd":2,"last update":"Sat Oct 21 05:14:26 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40000000000000002,"15min":0.53400000000000003},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.0289999999999999,"5min":1.782,"15min":19.559999999999999},"last":0.249},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.38400000000000001,"15min":0.49399999999999999},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.19800000000000001},"max":{"1min":0.55800000000000005,"5min":2.8029999999999999,"15min":19.521999999999998},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3943399424,"data_stored":4515856384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3801427968,"data_stored":4348305427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7745060864,"data_stored":8864395283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:15:08.724 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:08.724 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:08.725 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:08.725 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:08.726 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:15:28.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:15:29.278 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:15:29.279 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:15:29.327 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2196,"stamp":"2023-10-21T05:15:27.683297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27719044328,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1919039,"num_read_kb":7497229,"num_write":1898942,"num_write_kb":7594747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76542,"ondisk_log_size":76542,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19499644,"kb_used_data":15187428,"kb_used_omap":1459,"kb_used_meta":4310732,"kb_avail":263615876,"statfs":{"total":289910292480,"available":269942657024,"internally_reserved":0,"allocated":15551926272,"data_stored":17788881357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":4414190132},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,6,4,7,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13434880,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3482,"num_read_kb":13884,"num_write":3702,"num_write_kb":14808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002061"},"pg_stats":[{"pgid":"2.7","version":"83'9394","reported_seq":12974,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:21.187693+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:15:21.187693+0000","last_peered":"2023-10-21T05:15:21.187693+0000","last_clean":"2023-10-21T05:15:21.187693+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:15:21.187693+0000","last_undegraded":"2023-10-21T05:15:21.187693+0000","last_fullsized":"2023-10-21T05:15:21.187693+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9394,"log_dups_size":0,"ondisk_log_size":9394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1000,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:27.740544+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:14:27.740544+0000","last_peered":"2023-10-21T05:14:27.740544+0000","last_clean":"2023-10-21T05:14:27.740544+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:14:27.740544+0000","last_undegraded":"2023-10-21T05:14:27.740544+0000","last_fullsized":"2023-10-21T05:14:27.740544+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"83'15505","reported_seq":21301,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:21.188030+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:15:21.188030+0000","last_peered":"2023-10-21T05:15:21.188030+0000","last_clean":"2023-10-21T05:15:21.188030+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:15:21.188030+0000","last_undegraded":"2023-10-21T05:15:21.188030+0000","last_fullsized":"2023-10-21T05:15:21.188030+0000","mapping_epoch":12,"log_start":"75'5500","ondisk_log_start":"75'5500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"83'3212","reported_seq":5002,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:20.663095+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:15:20.663095+0000","last_peered":"2023-10-21T05:15:20.663095+0000","last_clean":"2023-10-21T05:15:20.663095+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:15:20.663095+0000","last_undegraded":"2023-10-21T05:15:20.663095+0000","last_fullsized":"2023-10-21T05:15:20.663095+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3212,"log_dups_size":0,"ondisk_log_size":3212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"83'479905","reported_seq":957012,"reported_epoch":83,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:23.566979+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:15:23.566979+0000","last_peered":"2023-10-21T05:15:23.566979+0000","last_clean":"2023-10-21T05:15:23.566979+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:15:23.566979+0000","last_undegraded":"2023-10-21T05:15:23.566979+0000","last_fullsized":"2023-10-21T05:15:23.566979+0000","mapping_epoch":15,"log_start":"81'469816","ondisk_log_start":"81'469816","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2224,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 257s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6966677504,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":478520,"num_read_kb":1883832,"num_write":476691,"num_write_kb":1906764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"83'15842","reported_seq":21862,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:22.193559+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:15:22.193559+0000","last_peered":"2023-10-21T05:15:22.193559+0000","last_clean":"2023-10-21T05:15:22.193559+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:15:22.193559+0000","last_undegraded":"2023-10-21T05:15:22.193559+0000","last_fullsized":"2023-10-21T05:15:22.193559+0000","mapping_epoch":12,"log_start":"75'5800","ondisk_log_start":"75'5800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"83'473576","reported_seq":943531,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:22.996117+0000","last_change":"2023-10-21T05:15:21.960548+0000","last_active":"2023-10-21T05:15:22.996117+0000","last_peered":"2023-10-21T05:15:22.996117+0000","last_clean":"2023-10-21T05:15:22.996117+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:15:22.996117+0000","last_undegraded":"2023-10-21T05:15:22.996117+0000","last_fullsized":"2023-10-21T05:15:22.996117+0000","mapping_epoch":15,"log_start":"81'463516","ondisk_log_start":"81'463516","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6872244224,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":471443,"num_read_kb":1855148,"num_write":471821,"num_write_kb":1887284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6277,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:21.194290+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:15:21.194290+0000","last_peered":"2023-10-21T05:15:21.194290+0000","last_clean":"2023-10-21T05:15:21.194290+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:15:21.194290+0000","last_undegraded":"2023-10-21T05:15:21.194290+0000","last_fullsized":"2023-10-21T05:15:21.194290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"83'493066","reported_seq":981500,"reported_epoch":83,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:23.560110+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:15:23.560110+0000","last_peered":"2023-10-21T05:15:23.560110+0000","last_clean":"2023-10-21T05:15:23.560110+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:15:23.560110+0000","last_undegraded":"2023-10-21T05:15:23.560110+0000","last_fullsized":"2023-10-21T05:15:23.560110+0000","mapping_epoch":15,"log_start":"81'483025","ondisk_log_start":"81'483025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2138,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 258s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7163441152,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":489920,"num_read_kb":1928660,"num_write":489843,"num_write_kb":1959372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":158,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:13:55.412993+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:13:55.412993+0000","last_peered":"2023-10-21T05:13:55.412993+0000","last_clean":"2023-10-21T05:13:55.412993+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:13:55.412993+0000","last_undegraded":"2023-10-21T05:13:55.412993+0000","last_fullsized":"2023-10-21T05:13:55.412993+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"83'462880","reported_seq":922517,"reported_epoch":83,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:23.009955+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:15:23.009955+0000","last_peered":"2023-10-21T05:15:23.009955+0000","last_clean":"2023-10-21T05:15:23.009955+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:15:23.009955+0000","last_undegraded":"2023-10-21T05:15:23.009955+0000","last_fullsized":"2023-10-21T05:15:23.009955+0000","mapping_epoch":15,"log_start":"81'452818","ondisk_log_start":"81'452818","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2133,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 253s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6716088339,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":460991,"num_read_kb":1814112,"num_write":459781,"num_write_kb":1839118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"83'3191","reported_seq":4516,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:20.671265+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:15:20.671265+0000","last_peered":"2023-10-21T05:15:20.671265+0000","last_clean":"2023-10-21T05:15:20.671265+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:15:20.671265+0000","last_undegraded":"2023-10-21T05:15:20.671265+0000","last_fullsized":"2023-10-21T05:15:20.671265+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3191,"log_dups_size":0,"ondisk_log_size":3191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":256,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:05.424423+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:15:05.424423+0000","last_peered":"2023-10-21T05:15:05.424423+0000","last_clean":"2023-10-21T05:15:05.424423+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:15:05.424423+0000","last_undegraded":"2023-10-21T05:15:05.424423+0000","last_fullsized":"2023-10-21T05:15:05.424423+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27718451219,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1900874,"num_read_kb":7481752,"num_write":1898136,"num_write_kb":7592538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15549784064,"data_stored":17788452902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40252,"ondisk_log_size":40252,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36191,"ondisk_log_size":36191,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739218,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5407452,"kb_used_data":3866900,"kb_used_omap":373,"kb_used_meta":1540170,"kb_avail":88964388,"statfs":{"total":96636764160,"available":91099533312,"internally_reserved":0,"allocated":3959705600,"data_stored":4531583359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1577134640},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,0,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:14:59 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.36499999999999999,"15min":0.40100000000000002},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":1.675,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.224},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.46400000000000002,"15min":0.44400000000000001},"min":{"1min":0.19500000000000001,"5min":0.191,"15min":0.187},"max":{"1min":2.02,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.20799999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:14:59 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.39600000000000002,"15min":0.53600000000000003},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":0.59999999999999998,"5min":2.488,"15min":14.318},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.439,"15min":0.59299999999999997},"min":{"1min":0.26900000000000002,"5min":0.25,"15min":0.20899999999999999},"max":{"1min":0.622,"5min":2.5209999999999999,"15min":14.343},"last":0.76800000000000002}]}]},{"osd":2,"up_from":8,"seq":34359739217,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8995952,"kb_used_data":7592680,"kb_used_omap":726,"kb_used_meta":1402537,"kb_avail":85375888,"statfs":{"total":96636764160,"available":87424909312,"internally_reserved":0,"allocated":7774904320,"data_stored":8893768551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":1436198882},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.433,"15min":0.45200000000000001},"min":{"1min":0.16600000000000001,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":4.3029999999999999,"5min":4.3029999999999999,"15min":5.8899999999999997},"last":1.6519999999999999},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.313,"15min":0.36899999999999999},"min":{"1min":0.20599999999999999,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":0.56299999999999994,"5min":0.60799999999999998,"15min":4.9029999999999996},"last":0.36499999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:14:56 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.437,"15min":0.48499999999999999},"min":{"1min":0.23499999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":1.8009999999999999,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.40400000000000003,"15min":0.57199999999999995},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.19400000000000001},"max":{"1min":1.766,"5min":2.8580000000000001,"15min":26.596},"last":0.33500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739216,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5096240,"kb_used_data":3727848,"kb_used_omap":359,"kb_used_meta":1368024,"kb_avail":89275600,"statfs":{"total":96636764160,"available":91418214400,"internally_reserved":0,"allocated":3817316352,"data_stored":4363529447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1400856610},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:26 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34300000000000003,"15min":0.40799999999999997},"min":{"1min":0.23699999999999999,"5min":0.16800000000000001,"15min":0.16400000000000001},"max":{"1min":0.624,"5min":3.1339999999999999,"15min":9.218},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.49199999999999999,"15min":0.51600000000000001},"min":{"1min":0.186,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.92600000000000005,"5min":5.6369999999999996,"15min":5.6390000000000002},"last":0.24199999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:14:26 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40000000000000002,"15min":0.53400000000000003},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.0289999999999999,"5min":1.782,"15min":19.559999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.38400000000000001,"15min":0.49399999999999999},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.19800000000000001},"max":{"1min":0.55800000000000005,"5min":2.8029999999999999,"15min":19.521999999999998},"last":0.29499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3958923264,"data_stored":4531380224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3815854080,"data_stored":4362731539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7775006720,"data_stored":8894341139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:15:29.329 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:29.329 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:29.329 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:29.330 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:29.330 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:15:29.858 INFO:teuthology.orchestra.run.smithi119.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-21T05:15:29.898 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:15:30.400 INFO:teuthology.orchestra.run.smithi119.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:15:30.447 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:15:31.074 INFO:teuthology.orchestra.run.smithi119.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-21T05:15:31.122 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:15:31.672 INFO:teuthology.orchestra.run.smithi119.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:15:31.704 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:15:33.904 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:33.897+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:15:33.904 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:33.897+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:15:33.929 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:33.925+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T05:15:33.929 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:33.925+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:15:33.929 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:33.925+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T05:15:33.930 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:33.925+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:15:33.930 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:33.925+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:15:33.930 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:33.925+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:15:33.930 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:33.925+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:15:51.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:15:52.205 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:15:52.206 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:15:52.236 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2208,"stamp":"2023-10-21T05:15:51.687622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27751353576,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1927487,"num_read_kb":7530965,"num_write":1907805,"num_write_kb":7630199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76471,"ondisk_log_size":76471,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19637464,"kb_used_data":15258684,"kb_used_omap":1459,"kb_used_meta":4377292,"kb_avail":263478056,"statfs":{"total":289910292480,"available":269801529344,"internally_reserved":0,"allocated":15624892416,"data_stored":17861847501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":4482347572},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,2,6,20,25,23],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12992512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3391,"num_read_kb":13532,"num_write":3556,"num_write_kb":14224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002097"},"pg_stats":[{"pgid":"2.7","version":"83'9457","reported_seq":13052,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:46.788618+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:15:46.788618+0000","last_peered":"2023-10-21T05:15:46.788618+0000","last_clean":"2023-10-21T05:15:46.788618+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:15:46.788618+0000","last_undegraded":"2023-10-21T05:15:46.788618+0000","last_fullsized":"2023-10-21T05:15:46.788618+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9457,"log_dups_size":0,"ondisk_log_size":9457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1000,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:27.740544+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:14:27.740544+0000","last_peered":"2023-10-21T05:14:27.740544+0000","last_clean":"2023-10-21T05:14:27.740544+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:14:27.740544+0000","last_undegraded":"2023-10-21T05:14:27.740544+0000","last_fullsized":"2023-10-21T05:14:27.740544+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"83'15610","reported_seq":21431,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:46.789153+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:15:46.789153+0000","last_peered":"2023-10-21T05:15:46.789153+0000","last_clean":"2023-10-21T05:15:46.789153+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:15:46.789153+0000","last_undegraded":"2023-10-21T05:15:46.789153+0000","last_fullsized":"2023-10-21T05:15:46.789153+0000","mapping_epoch":12,"log_start":"75'5600","ondisk_log_start":"75'5600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"83'3233","reported_seq":5028,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:45.989597+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:15:45.989597+0000","last_peered":"2023-10-21T05:15:45.989597+0000","last_clean":"2023-10-21T05:15:45.989597+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:15:45.989597+0000","last_undegraded":"2023-10-21T05:15:45.989597+0000","last_fullsized":"2023-10-21T05:15:45.989597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3233,"log_dups_size":0,"ondisk_log_size":3233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"83'482247","reported_seq":961495,"reported_epoch":83,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:48.701220+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:15:48.701220+0000","last_peered":"2023-10-21T05:15:48.701220+0000","last_clean":"2023-10-21T05:15:48.701220+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:15:48.701220+0000","last_undegraded":"2023-10-21T05:15:48.701220+0000","last_fullsized":"2023-10-21T05:15:48.701220+0000","mapping_epoch":15,"log_start":"82'472216","ondisk_log_start":"82'472216","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2441,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 282s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6974685184,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":480640,"num_read_kb":1892296,"num_write":478896,"num_write_kb":1915584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"83'15902","reported_seq":21947,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:48.411442+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:15:48.411442+0000","last_peered":"2023-10-21T05:15:48.411442+0000","last_clean":"2023-10-21T05:15:48.411442+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:15:48.411442+0000","last_undegraded":"2023-10-21T05:15:48.411442+0000","last_fullsized":"2023-10-21T05:15:48.411442+0000","mapping_epoch":12,"log_start":"75'5900","ondisk_log_start":"75'5900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"83'475829","reported_seq":947934,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:48.089880+0000","last_change":"2023-10-21T05:15:43.752091+0000","last_active":"2023-10-21T05:15:48.089880+0000","last_peered":"2023-10-21T05:15:48.089880+0000","last_clean":"2023-10-21T05:15:48.089880+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:15:48.089880+0000","last_undegraded":"2023-10-21T05:15:48.089880+0000","last_fullsized":"2023-10-21T05:15:48.089880+0000","mapping_epoch":15,"log_start":"82'465816","ondisk_log_start":"82'465816","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6880313344,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":473583,"num_read_kb":1863700,"num_write":474074,"num_write_kb":1896296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6302,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:46.788837+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:15:46.788837+0000","last_peered":"2023-10-21T05:15:46.788837+0000","last_clean":"2023-10-21T05:15:46.788837+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:15:46.788837+0000","last_undegraded":"2023-10-21T05:15:46.788837+0000","last_fullsized":"2023-10-21T05:15:46.788837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"83'495448","reported_seq":986031,"reported_epoch":83,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:48.689086+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:15:48.689086+0000","last_peered":"2023-10-21T05:15:48.689086+0000","last_clean":"2023-10-21T05:15:48.689086+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:15:48.689086+0000","last_undegraded":"2023-10-21T05:15:48.689086+0000","last_fullsized":"2023-10-21T05:15:48.689086+0000","mapping_epoch":15,"log_start":"82'485425","ondisk_log_start":"82'485425","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2351,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 283s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7171756032,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":492049,"num_read_kb":1937152,"num_write":492091,"num_write_kb":1968364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":159,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:05.437518+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:15:05.437518+0000","last_peered":"2023-10-21T05:15:05.437518+0000","last_clean":"2023-10-21T05:15:05.437518+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:15:05.437518+0000","last_undegraded":"2023-10-21T05:15:05.437518+0000","last_fullsized":"2023-10-21T05:15:05.437518+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"83'465162","reported_seq":926882,"reported_epoch":83,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:48.104812+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:15:48.104812+0000","last_peered":"2023-10-21T05:15:48.104812+0000","last_clean":"2023-10-21T05:15:48.104812+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:15:48.104812+0000","last_undegraded":"2023-10-21T05:15:48.104812+0000","last_fullsized":"2023-10-21T05:15:48.104812+0000","mapping_epoch":15,"log_start":"82'455118","ondisk_log_start":"82'455118","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2334,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 278s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6724005907,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":463050,"num_read_kb":1822340,"num_write":461938,"num_write_kb":1847746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"83'3212","reported_seq":4542,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:46.043551+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:15:46.043551+0000","last_peered":"2023-10-21T05:15:46.043551+0000","last_clean":"2023-10-21T05:15:46.043551+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:15:46.043551+0000","last_undegraded":"2023-10-21T05:15:46.043551+0000","last_fullsized":"2023-10-21T05:15:46.043551+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3212,"log_dups_size":0,"ondisk_log_size":3212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":256,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:05.424423+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:15:05.424423+0000","last_peered":"2023-10-21T05:15:05.424423+0000","last_clean":"2023-10-21T05:15:05.424423+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:15:05.424423+0000","last_undegraded":"2023-10-21T05:15:05.424423+0000","last_fullsized":"2023-10-21T05:15:05.424423+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27750760467,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1909322,"num_read_kb":7515488,"num_write":1906999,"num_write_kb":7627990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15622471680,"data_stored":17861140518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40111,"ondisk_log_size":40111,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36261,"ondisk_log_size":36261,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739223,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5474180,"kb_used_data":3885372,"kb_used_omap":373,"kb_used_meta":1588426,"kb_avail":88897660,"statfs":{"total":96636764160,"available":91031203840,"internally_reserved":0,"allocated":3978620928,"data_stored":4550498687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1626548784},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,0,4,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:14:59 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.36499999999999999,"15min":0.40100000000000002},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":1.675,"5min":4.9889999999999999,"15min":5.9009999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.46400000000000002,"15min":0.44400000000000001},"min":{"1min":0.19500000000000001,"5min":0.191,"15min":0.187},"max":{"1min":2.02,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.21199999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:14:59 2023","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.39600000000000002,"15min":0.53600000000000003},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":0.59999999999999998,"5min":2.488,"15min":14.318},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.439,"15min":0.59299999999999997},"min":{"1min":0.26900000000000002,"5min":0.25,"15min":0.20899999999999999},"max":{"1min":0.622,"5min":2.5209999999999999,"15min":14.343},"last":0.307}]}]},{"osd":2,"up_from":8,"seq":34359739222,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8995500,"kb_used_data":7628064,"kb_used_omap":726,"kb_used_meta":1366697,"kb_avail":85376340,"statfs":{"total":96636764160,"available":87425372160,"internally_reserved":0,"allocated":7811137536,"data_stored":8930001767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":1399498722},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,5,6,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.433,"15min":0.45200000000000001},"min":{"1min":0.16600000000000001,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":4.3029999999999999,"5min":4.3029999999999999,"15min":5.8899999999999997},"last":0.215},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.313,"15min":0.36899999999999999},"min":{"1min":0.20599999999999999,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":0.56299999999999994,"5min":0.60799999999999998,"15min":4.9029999999999996},"last":0.38200000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:14:56 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.437,"15min":0.48499999999999999},"min":{"1min":0.23499999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":1.8009999999999999,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.40400000000000003,"15min":0.57199999999999995},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.19400000000000001},"max":{"1min":1.766,"5min":2.8580000000000001,"15min":26.596},"last":0.34999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739221,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5167784,"kb_used_data":3745248,"kb_used_omap":359,"kb_used_meta":1422168,"kb_avail":89204056,"statfs":{"total":96636764160,"available":91344953344,"internally_reserved":0,"allocated":3835133952,"data_stored":4381347047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1456300066},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,11,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.372,"15min":0.40400000000000003},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16400000000000001},"max":{"1min":2.613,"5min":3.1339999999999999,"15min":9.218},"last":0.307},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.437,"15min":0.50600000000000001},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":1.1579999999999999,"5min":5.6369999999999996,"15min":5.6390000000000002},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.38800000000000001,"15min":0.53400000000000003},"min":{"1min":0.215,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.665,"5min":2.665,"15min":19.559999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.39500000000000002,"15min":0.48699999999999999},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.19800000000000001},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":19.521999999999998},"last":0.34000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3977175040,"data_stored":4549632000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3833925632,"data_stored":4380803091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7811371008,"data_stored":8930705427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:15:52.238 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:52.239 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:52.239 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:52.240 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:52.240 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:15:54.715 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:54.701+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:15:54.715 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:54.701+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:15:54.715 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:54.705+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:15:54.716 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:54.705+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:15:54.716 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:54.705+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:15:54.716 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:54.705+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:15:54.716 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:15:54.709+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:16:12.241 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:16:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:16:12.697 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:16:12.724 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2218,"stamp":"2023-10-21T05:16:11.693192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27777920232,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1934353,"num_read_kb":7558373,"num_write":1915040,"num_write_kb":7659139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76683,"ondisk_log_size":76683,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19794404,"kb_used_data":15315660,"kb_used_omap":1459,"kb_used_meta":4477260,"kb_avail":263321116,"statfs":{"total":289910292480,"available":269640822784,"internally_reserved":0,"allocated":15683235840,"data_stored":17920190925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":4584714804},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,10,12,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13402112,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3439,"num_read_kb":13724,"num_write":3645,"num_write_kb":14580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004193"},"pg_stats":[{"pgid":"2.7","version":"83'9487","reported_seq":13094,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:16:07.271902+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:16:07.271902+0000","last_peered":"2023-10-21T05:16:07.271902+0000","last_clean":"2023-10-21T05:16:07.271902+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:16:07.271902+0000","last_undegraded":"2023-10-21T05:16:07.271902+0000","last_fullsized":"2023-10-21T05:16:07.271902+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1000,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-21T05:14:27.740544+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:14:27.740544+0000","last_peered":"2023-10-21T05:14:27.740544+0000","last_clean":"2023-10-21T05:14:27.740544+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:14:27.740544+0000","last_undegraded":"2023-10-21T05:14:27.740544+0000","last_fullsized":"2023-10-21T05:14:27.740544+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"83'15657","reported_seq":21498,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:16:07.272787+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:16:07.272787+0000","last_peered":"2023-10-21T05:16:07.272787+0000","last_clean":"2023-10-21T05:16:07.272787+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:16:07.272787+0000","last_undegraded":"2023-10-21T05:16:07.272787+0000","last_fullsized":"2023-10-21T05:16:07.272787+0000","mapping_epoch":12,"log_start":"75'5600","ondisk_log_start":"75'5600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"83'3243","reported_seq":5042,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:16:06.163494+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:16:06.163494+0000","last_peered":"2023-10-21T05:16:06.163494+0000","last_clean":"2023-10-21T05:16:06.163494+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:16:06.163494+0000","last_undegraded":"2023-10-21T05:16:06.163494+0000","last_fullsized":"2023-10-21T05:16:06.163494+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"83'484161","reported_seq":965178,"reported_epoch":83,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:08.823768+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:16:08.823768+0000","last_peered":"2023-10-21T05:16:08.823768+0000","last_clean":"2023-10-21T05:16:08.823768+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:16:08.823768+0000","last_undegraded":"2023-10-21T05:16:08.823768+0000","last_fullsized":"2023-10-21T05:16:08.823768+0000","mapping_epoch":15,"log_start":"82'474116","ondisk_log_start":"82'474116","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2622,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 302s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6981210112,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":482395,"num_read_kb":1899296,"num_write":480691,"num_write_kb":1922764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"83'15957","reported_seq":22022,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:16:07.937139+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:16:07.937139+0000","last_peered":"2023-10-21T05:16:07.937139+0000","last_clean":"2023-10-21T05:16:07.937139+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:16:07.937139+0000","last_undegraded":"2023-10-21T05:16:07.937139+0000","last_fullsized":"2023-10-21T05:16:07.937139+0000","mapping_epoch":12,"log_start":"75'5900","ondisk_log_start":"75'5900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"83'477633","reported_seq":951357,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:16:08.181228+0000","last_change":"2023-10-21T05:16:06.881402+0000","last_active":"2023-10-21T05:16:08.181228+0000","last_peered":"2023-10-21T05:16:08.181228+0000","last_clean":"2023-10-21T05:16:08.181228+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:16:08.181228+0000","last_undegraded":"2023-10-21T05:16:08.181228+0000","last_fullsized":"2023-10-21T05:16:08.181228+0000","mapping_epoch":15,"log_start":"82'467616","ondisk_log_start":"82'467616","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6886952960,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":475196,"num_read_kb":1870144,"num_write":475878,"num_write_kb":1903512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6323,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:16:07.275158+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:16:07.275158+0000","last_peered":"2023-10-21T05:16:07.275158+0000","last_clean":"2023-10-21T05:16:07.275158+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:16:07.275158+0000","last_undegraded":"2023-10-21T05:16:07.275158+0000","last_fullsized":"2023-10-21T05:16:07.275158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"83'497394","reported_seq":989787,"reported_epoch":83,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:08.823301+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:16:08.823301+0000","last_peered":"2023-10-21T05:16:08.823301+0000","last_clean":"2023-10-21T05:16:08.823301+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:16:08.823301+0000","last_undegraded":"2023-10-21T05:16:08.823301+0000","last_fullsized":"2023-10-21T05:16:08.823301+0000","mapping_epoch":15,"log_start":"82'487325","ondisk_log_start":"82'487325","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2471,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 303s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7178739712,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":493847,"num_read_kb":1944332,"num_write":493962,"num_write_kb":1975848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":159,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:05.437518+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:15:05.437518+0000","last_peered":"2023-10-21T05:15:05.437518+0000","last_clean":"2023-10-21T05:15:05.437518+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:15:05.437518+0000","last_undegraded":"2023-10-21T05:15:05.437518+0000","last_fullsized":"2023-10-21T05:15:05.437518+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"83'467058","reported_seq":930491,"reported_epoch":83,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:08.271071+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:16:08.271071+0000","last_peered":"2023-10-21T05:16:08.271071+0000","last_clean":"2023-10-21T05:16:08.271071+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:16:08.271071+0000","last_undegraded":"2023-10-21T05:16:08.271071+0000","last_fullsized":"2023-10-21T05:16:08.271071+0000","mapping_epoch":15,"log_start":"82'457018","ondisk_log_start":"82'457018","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2537,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 298s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6730424339,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":464750,"num_read_kb":1829124,"num_write":463703,"num_write_kb":1854806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"83'3222","reported_seq":4556,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:16:06.191122+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:16:06.191122+0000","last_peered":"2023-10-21T05:16:06.191122+0000","last_clean":"2023-10-21T05:16:06.191122+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:16:06.191122+0000","last_undegraded":"2023-10-21T05:16:06.191122+0000","last_fullsized":"2023-10-21T05:16:06.191122+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3222,"log_dups_size":0,"ondisk_log_size":3222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":256,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:05.424423+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:15:05.424423+0000","last_peered":"2023-10-21T05:15:05.424423+0000","last_clean":"2023-10-21T05:15:05.424423+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:15:05.424423+0000","last_undegraded":"2023-10-21T05:15:05.424423+0000","last_fullsized":"2023-10-21T05:15:05.424423+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27777327123,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1916188,"num_read_kb":7542896,"num_write":1914234,"num_write_kb":7656930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15681740800,"data_stored":17920409638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40171,"ondisk_log_size":40171,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36413,"ondisk_log_size":36413,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739227,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5524276,"kb_used_data":3899372,"kb_used_omap":373,"kb_used_meta":1624522,"kb_avail":88847564,"statfs":{"total":96636764160,"available":90979905536,"internally_reserved":0,"allocated":3992956928,"data_stored":4564834687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1663511088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,7,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:16:01 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.34100000000000003,"15min":0.40899999999999997},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":2.964,"5min":2.964,"15min":5.9009999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.42399999999999999,"15min":0.433},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.8170000000000002,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.23200000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:16:01 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.35999999999999999,"15min":0.50600000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.17899999999999999},"max":{"1min":0.72899999999999998,"5min":2.1219999999999999,"15min":14.318},"last":3.8290000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.40600000000000003,"15min":0.55500000000000005},"min":{"1min":0.27800000000000002,"5min":0.26200000000000001,"15min":0.20899999999999999},"max":{"1min":0.76800000000000002,"5min":2.3050000000000002,"15min":14.343},"last":3.8730000000000002}]}]},{"osd":2,"up_from":8,"seq":34359739226,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9069952,"kb_used_data":7656696,"kb_used_omap":726,"kb_used_meta":1412521,"kb_avail":85301888,"statfs":{"total":96636764160,"available":87349133312,"internally_reserved":0,"allocated":7840456704,"data_stored":8959320935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":1446422498},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40799999999999997,"15min":0.45800000000000002},"min":{"1min":0.16800000000000001,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":1.6519999999999999,"5min":4.3029999999999999,"15min":5.8899999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.29599999999999999,"5min":0.315,"15min":0.35999999999999999},"min":{"1min":0.22500000000000001,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":0.49199999999999999,"5min":0.60799999999999998,"15min":4.9029999999999996},"last":0.46600000000000003}]},{"osd":1,"last update":"Sat Oct 21 05:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48799999999999999,"15min":0.501},"min":{"1min":0.24099999999999999,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":3.8140000000000001,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.44900000000000001,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.19400000000000001},"max":{"1min":3.8370000000000002,"5min":3.8370000000000002,"15min":26.596},"last":0.249}]}]},{"osd":1,"up_from":8,"seq":34359739225,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5200176,"kb_used_data":3759592,"kb_used_omap":359,"kb_used_meta":1440216,"kb_avail":89171664,"statfs":{"total":96636764160,"available":91311783936,"internally_reserved":0,"allocated":3849822208,"data_stored":4396035303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1474781218},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,4,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.372,"15min":0.40400000000000003},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16400000000000001},"max":{"1min":2.613,"5min":3.1339999999999999,"15min":9.218},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.437,"15min":0.50600000000000001},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":1.1579999999999999,"5min":5.6369999999999996,"15min":5.6390000000000002},"last":0.23999999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.38800000000000001,"15min":0.53400000000000003},"min":{"1min":0.215,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.665,"5min":2.665,"15min":19.559999999999999},"last":0.218},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.39500000000000002,"15min":0.48699999999999999},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.19800000000000001},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":19.521999999999998},"last":0.26100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3992178688,"data_stored":4564635648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3848560640,"data_stored":4395438099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7841001472,"data_stored":8960335891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:16:12.726 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:12.727 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:12.727 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:12.728 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:12.729 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:16:32.730 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:16:33.367 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:16:33.368 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:16:33.433 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2229,"stamp":"2023-10-21T05:16:31.696573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27804400872,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1941445,"num_read_kb":7586681,"num_write":1922315,"num_write_kb":7688239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76791,"ondisk_log_size":76791,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19999304,"kb_used_data":15374320,"kb_used_omap":1459,"kb_used_meta":4623500,"kb_avail":263116216,"statfs":{"total":289910292480,"available":269431005184,"internally_reserved":0,"allocated":15743303680,"data_stored":17980245433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":4734464564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,7,5,7,19,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13135872,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3563,"num_read_kb":14216,"num_write":3616,"num_write_kb":14464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002124"},"pg_stats":[{"pgid":"2.7","version":"84'9517","reported_seq":13137,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:27.814656+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:16:27.814656+0000","last_peered":"2023-10-21T05:16:27.814656+0000","last_clean":"2023-10-21T05:16:27.814656+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:16:27.814656+0000","last_undegraded":"2023-10-21T05:16:27.814656+0000","last_fullsized":"2023-10-21T05:16:27.814656+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1001,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:05.424381+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:15:05.424381+0000","last_peered":"2023-10-21T05:15:05.424381+0000","last_clean":"2023-10-21T05:15:05.424381+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:15:05.424381+0000","last_undegraded":"2023-10-21T05:15:05.424381+0000","last_fullsized":"2023-10-21T05:15:05.424381+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"84'15707","reported_seq":21570,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:27.825572+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:16:27.825572+0000","last_peered":"2023-10-21T05:16:27.825572+0000","last_clean":"2023-10-21T05:16:27.825572+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:16:27.825572+0000","last_undegraded":"2023-10-21T05:16:27.825572+0000","last_fullsized":"2023-10-21T05:16:27.825572+0000","mapping_epoch":12,"log_start":"75'5700","ondisk_log_start":"75'5700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"84'3253","reported_seq":5058,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:27.751660+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:16:27.751660+0000","last_peered":"2023-10-21T05:16:27.751660+0000","last_clean":"2023-10-21T05:16:27.751660+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:16:27.751660+0000","last_undegraded":"2023-10-21T05:16:27.751660+0000","last_fullsized":"2023-10-21T05:16:27.751660+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3253,"log_dups_size":0,"ondisk_log_size":3253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"84'486108","reported_seq":968916,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:28.908284+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:16:28.908284+0000","last_peered":"2023-10-21T05:16:28.908284+0000","last_clean":"2023-10-21T05:16:28.908284+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:16:28.908284+0000","last_undegraded":"2023-10-21T05:16:28.908284+0000","last_fullsized":"2023-10-21T05:16:28.908284+0000","mapping_epoch":15,"log_start":"82'476016","ondisk_log_start":"82'476016","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2788,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 322s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6987939840,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":484168,"num_read_kb":1906380,"num_write":482533,"num_write_kb":1930132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'15982","reported_seq":22069,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:27.752087+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:16:27.752087+0000","last_peered":"2023-10-21T05:16:27.752087+0000","last_clean":"2023-10-21T05:16:27.752087+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:16:27.752087+0000","last_undegraded":"2023-10-21T05:16:27.752087+0000","last_fullsized":"2023-10-21T05:16:27.752087+0000","mapping_epoch":12,"log_start":"75'5900","ondisk_log_start":"75'5900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"84'479423","reported_seq":954893,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:28.321165+0000","last_change":"2023-10-21T05:16:25.165758+0000","last_active":"2023-10-21T05:16:28.321165+0000","last_peered":"2023-10-21T05:16:28.321165+0000","last_clean":"2023-10-21T05:16:28.321165+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:16:28.321165+0000","last_undegraded":"2023-10-21T05:16:28.321165+0000","last_fullsized":"2023-10-21T05:16:28.321165+0000","mapping_epoch":15,"log_start":"82'469416","ondisk_log_start":"82'469416","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6893375488,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":476930,"num_read_kb":1877068,"num_write":477668,"num_write_kb":1910672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6344,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:27.756247+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:16:27.756247+0000","last_peered":"2023-10-21T05:16:27.756247+0000","last_clean":"2023-10-21T05:16:27.756247+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:16:27.756247+0000","last_undegraded":"2023-10-21T05:16:27.756247+0000","last_fullsized":"2023-10-21T05:16:27.756247+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"84'499317","reported_seq":993547,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:28.914694+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:16:28.914694+0000","last_peered":"2023-10-21T05:16:28.914694+0000","last_clean":"2023-10-21T05:16:28.914694+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:16:28.914694+0000","last_undegraded":"2023-10-21T05:16:28.914694+0000","last_fullsized":"2023-10-21T05:16:28.914694+0000","mapping_epoch":15,"log_start":"82'489225","ondisk_log_start":"82'489225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2643,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 323s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7185367040,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":495661,"num_read_kb":1951560,"num_write":495783,"num_write_kb":1983132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":160,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:28.486629+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:15:28.486629+0000","last_peered":"2023-10-21T05:15:28.486629+0000","last_clean":"2023-10-21T05:15:28.486629+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:15:28.486629+0000","last_undegraded":"2023-10-21T05:15:28.486629+0000","last_fullsized":"2023-10-21T05:15:28.486629+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"84'468981","reported_seq":934200,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:28.325577+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:16:28.325577+0000","last_peered":"2023-10-21T05:16:28.325577+0000","last_clean":"2023-10-21T05:16:28.325577+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:16:28.325577+0000","last_undegraded":"2023-10-21T05:16:28.325577+0000","last_fullsized":"2023-10-21T05:16:28.325577+0000","mapping_epoch":15,"log_start":"82'458918","ondisk_log_start":"82'458918","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2701,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 318s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6737125395,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":466521,"num_read_kb":1836196,"num_write":465525,"num_write_kb":1862094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"84'3232","reported_seq":4572,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:27.751327+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:16:27.751327+0000","last_peered":"2023-10-21T05:16:27.751327+0000","last_clean":"2023-10-21T05:16:27.751327+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:16:27.751327+0000","last_undegraded":"2023-10-21T05:16:27.751327+0000","last_fullsized":"2023-10-21T05:16:27.751327+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3232,"log_dups_size":0,"ondisk_log_size":3232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":257,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:23.008635+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:15:23.008635+0000","last_peered":"2023-10-21T05:15:23.008635+0000","last_clean":"2023-10-21T05:15:23.008635+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:15:23.008635+0000","last_undegraded":"2023-10-21T05:15:23.008635+0000","last_fullsized":"2023-10-21T05:15:23.008635+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27803807763,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1923280,"num_read_kb":7571204,"num_write":1921509,"num_write_kb":7686030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15741276160,"data_stored":17979944998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40254,"ondisk_log_size":40254,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36438,"ondisk_log_size":36438,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739231,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5575700,"kb_used_data":3914700,"kb_used_omap":373,"kb_used_meta":1660618,"kb_avail":88796140,"statfs":{"total":96636764160,"available":90927247360,"internally_reserved":0,"allocated":4008652800,"data_stored":4580526115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1700473392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,0,2,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:16:01 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.34100000000000003,"15min":0.40899999999999997},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":2.964,"5min":2.964,"15min":5.9009999999999998},"last":4.0259999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.42399999999999999,"15min":0.433},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.8170000000000002,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":4.2530000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:16:01 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.35999999999999999,"15min":0.50600000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.17899999999999999},"max":{"1min":0.72899999999999998,"5min":2.1219999999999999,"15min":14.318},"last":4.2990000000000004},{"interface":"front","average":{"1min":0.379,"5min":0.40600000000000003,"15min":0.55500000000000005},"min":{"1min":0.27800000000000002,"5min":0.26200000000000001,"15min":0.20899999999999999},"max":{"1min":0.76800000000000002,"5min":2.3050000000000002,"15min":14.343},"last":4.3230000000000004}]}]},{"osd":2,"up_from":8,"seq":34359739230,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9166860,"kb_used_data":7685572,"kb_used_omap":726,"kb_used_meta":1480553,"kb_avail":85204980,"statfs":{"total":96636764160,"available":87249899520,"internally_reserved":0,"allocated":7870025728,"data_stored":8988885515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":1516087266},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40799999999999997,"15min":0.45800000000000002},"min":{"1min":0.16800000000000001,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":1.6519999999999999,"5min":4.3029999999999999,"15min":5.8899999999999997},"last":0.17799999999999999},{"interface":"front","average":{"1min":0.29599999999999999,"5min":0.315,"15min":0.35999999999999999},"min":{"1min":0.22500000000000001,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":0.49199999999999999,"5min":0.60799999999999998,"15min":4.9029999999999996},"last":0.223}]},{"osd":1,"last update":"Sat Oct 21 05:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48799999999999999,"15min":0.501},"min":{"1min":0.24099999999999999,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":3.8140000000000001,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.44900000000000001,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.19400000000000001},"max":{"1min":3.8370000000000002,"5min":3.8370000000000002,"15min":26.596},"last":0.38500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739229,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5256744,"kb_used_data":3774048,"kb_used_omap":359,"kb_used_meta":1482328,"kb_avail":89115096,"statfs":{"total":96636764160,"available":91253858304,"internally_reserved":0,"allocated":3864625152,"data_stored":4410833803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1517903906},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,4,12,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.372,"15min":0.40400000000000003},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16400000000000001},"max":{"1min":2.613,"5min":3.1339999999999999,"15min":9.218},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.437,"15min":0.50600000000000001},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":1.1579999999999999,"5min":5.6369999999999996,"15min":5.6390000000000002},"last":0.35799999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.38800000000000001,"15min":0.53400000000000003},"min":{"1min":0.215,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.665,"5min":2.665,"15min":19.559999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.39500000000000002,"15min":0.48699999999999999},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.19800000000000001},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":19.521999999999998},"last":0.34100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4007182336,"data_stored":4579639296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3863318528,"data_stored":4410195987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7870775296,"data_stored":8990109715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:16:33.435 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:33.436 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:33.436 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:33.436 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:33.437 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:16:53.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:16:53.894 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:16:53.895 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:16:53.924 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2240,"stamp":"2023-10-21T05:16:53.700410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27833638120,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1949546,"num_read_kb":7619021,"num_write":1930347,"num_write_kb":7720367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76933,"ondisk_log_size":76933,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20142656,"kb_used_data":15435688,"kb_used_omap":1459,"kb_used_meta":4705484,"kb_avail":262972864,"statfs":{"total":289910292480,"available":269284212736,"internally_reserved":0,"allocated":15806144512,"data_stored":18043086265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":4818416180},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,14,8,8,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16166912,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4487,"num_read_kb":17920,"num_write":4433,"num_write_kb":17732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002150"},"pg_stats":[{"pgid":"2.7","version":"84'9559","reported_seq":13195,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:53.348128+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:16:53.348128+0000","last_peered":"2023-10-21T05:16:53.348128+0000","last_clean":"2023-10-21T05:16:53.348128+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:16:53.348128+0000","last_undegraded":"2023-10-21T05:16:53.348128+0000","last_fullsized":"2023-10-21T05:16:53.348128+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1002,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:15.433536+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:16:15.433536+0000","last_peered":"2023-10-21T05:16:15.433536+0000","last_clean":"2023-10-21T05:16:15.433536+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:16:15.433536+0000","last_undegraded":"2023-10-21T05:16:15.433536+0000","last_fullsized":"2023-10-21T05:16:15.433536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"84'15777","reported_seq":21665,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:53.348517+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:16:53.348517+0000","last_peered":"2023-10-21T05:16:53.348517+0000","last_clean":"2023-10-21T05:16:53.348517+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:16:53.348517+0000","last_undegraded":"2023-10-21T05:16:53.348517+0000","last_fullsized":"2023-10-21T05:16:53.348517+0000","mapping_epoch":12,"log_start":"75'5700","ondisk_log_start":"75'5700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"84'3267","reported_seq":5077,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:52.585516+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:16:52.585516+0000","last_peered":"2023-10-21T05:16:52.585516+0000","last_clean":"2023-10-21T05:16:52.585516+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:16:52.585516+0000","last_undegraded":"2023-10-21T05:16:52.585516+0000","last_fullsized":"2023-10-21T05:16:52.585516+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3267,"log_dups_size":0,"ondisk_log_size":3267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"84'488001","reported_seq":972674,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:49.011357+0000","last_change":"2023-10-21T05:11:06.532156+0000","last_active":"2023-10-21T05:16:49.011357+0000","last_peered":"2023-10-21T05:16:49.011357+0000","last_clean":"2023-10-21T05:16:49.011357+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:16:49.011357+0000","last_undegraded":"2023-10-21T05:16:49.011357+0000","last_fullsized":"2023-10-21T05:16:49.011357+0000","mapping_epoch":15,"log_start":"82'477917","ondisk_log_start":"82'477917","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2926,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 342s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":6994526208,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":486015,"num_read_kb":1913756,"num_write":484344,"num_write_kb":1937376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'16042","reported_seq":22149,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:48.227940+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:16:48.227940+0000","last_peered":"2023-10-21T05:16:48.227940+0000","last_clean":"2023-10-21T05:16:48.227940+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:16:48.227940+0000","last_undegraded":"2023-10-21T05:16:48.227940+0000","last_fullsized":"2023-10-21T05:16:48.227940+0000","mapping_epoch":12,"log_start":"75'6000","ondisk_log_start":"75'6000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"84'481613","reported_seq":959318,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:53.408607+0000","last_change":"2023-10-21T05:16:51.933433+0000","last_active":"2023-10-21T05:16:53.408607+0000","last_peered":"2023-10-21T05:16:53.408607+0000","last_clean":"2023-10-21T05:16:53.408607+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:16:53.408607+0000","last_undegraded":"2023-10-21T05:16:53.408607+0000","last_fullsized":"2023-10-21T05:16:53.408607+0000","mapping_epoch":15,"log_start":"82'471516","ondisk_log_start":"82'471516","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6901374976,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":479158,"num_read_kb":1885964,"num_write":479858,"num_write_kb":1919432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6364,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:48.229598+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:16:48.229598+0000","last_peered":"2023-10-21T05:16:48.229598+0000","last_clean":"2023-10-21T05:16:48.229598+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:16:48.229598+0000","last_undegraded":"2023-10-21T05:16:48.229598+0000","last_fullsized":"2023-10-21T05:16:48.229598+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"84'501303","reported_seq":997371,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:49.007425+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:16:49.007425+0000","last_peered":"2023-10-21T05:16:49.007425+0000","last_clean":"2023-10-21T05:16:49.007425+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:16:49.007425+0000","last_undegraded":"2023-10-21T05:16:49.007425+0000","last_fullsized":"2023-10-21T05:16:49.007425+0000","mapping_epoch":15,"log_start":"82'491225","ondisk_log_start":"82'491225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2795,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 343s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7192211456,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":497489,"num_read_kb":1958864,"num_write":497671,"num_write_kb":1990684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":160,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:28.486629+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:15:28.486629+0000","last_peered":"2023-10-21T05:15:28.486629+0000","last_clean":"2023-10-21T05:15:28.486629+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:15:28.486629+0000","last_undegraded":"2023-10-21T05:15:28.486629+0000","last_fullsized":"2023-10-21T05:15:28.486629+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"84'471255","reported_seq":938696,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:53.420714+0000","last_change":"2023-10-21T05:11:09.744175+0000","last_active":"2023-10-21T05:16:53.420714+0000","last_peered":"2023-10-21T05:16:53.420714+0000","last_clean":"2023-10-21T05:16:53.420714+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:16:53.420714+0000","last_undegraded":"2023-10-21T05:16:53.420714+0000","last_fullsized":"2023-10-21T05:16:53.420714+0000","mapping_epoch":15,"log_start":"82'461218","ondisk_log_start":"82'461218","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2915,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 343s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6744932371,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":468719,"num_read_kb":1844960,"num_write":467668,"num_write_kb":1870666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"84'3246","reported_seq":4591,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:52.619376+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:16:52.619376+0000","last_peered":"2023-10-21T05:16:52.619376+0000","last_clean":"2023-10-21T05:16:52.619376+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:16:52.619376+0000","last_undegraded":"2023-10-21T05:16:52.619376+0000","last_fullsized":"2023-10-21T05:16:52.619376+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3246,"log_dups_size":0,"ondisk_log_size":3246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":257,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:23.008635+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:15:23.008635+0000","last_peered":"2023-10-21T05:15:23.008635+0000","last_clean":"2023-10-21T05:15:23.008635+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:15:23.008635+0000","last_undegraded":"2023-10-21T05:15:23.008635+0000","last_fullsized":"2023-10-21T05:15:23.008635+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27833045011,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1931381,"num_read_kb":7603544,"num_write":1929541,"num_write_kb":7718158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15803482112,"data_stored":18042150950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40296,"ondisk_log_size":40296,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36538,"ondisk_log_size":36538,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739235,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5607868,"kb_used_data":3928820,"kb_used_omap":373,"kb_used_meta":1678666,"kb_avail":88763972,"statfs":{"total":96636764160,"available":90894307328,"internally_reserved":0,"allocated":4023111680,"data_stored":4594984995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1718954544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,9,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:16:01 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.34100000000000003,"15min":0.40899999999999997},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":2.964,"5min":2.964,"15min":5.9009999999999998},"last":0.373},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.42399999999999999,"15min":0.433},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.8170000000000002,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:16:01 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.35999999999999999,"15min":0.50600000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.17899999999999999},"max":{"1min":0.72899999999999998,"5min":2.1219999999999999,"15min":14.318},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.40600000000000003,"15min":0.55500000000000005},"min":{"1min":0.27800000000000002,"5min":0.26200000000000001,"15min":0.20899999999999999},"max":{"1min":0.76800000000000002,"5min":2.3050000000000002,"15min":14.343},"last":0.42699999999999999}]}]},{"osd":2,"up_from":8,"seq":34359739234,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9200288,"kb_used_data":7715608,"kb_used_omap":726,"kb_used_meta":1483945,"kb_avail":85171552,"statfs":{"total":96636764160,"available":87215669248,"internally_reserved":0,"allocated":7900782592,"data_stored":9019642379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":1519560674},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40799999999999997,"15min":0.45800000000000002},"min":{"1min":0.16800000000000001,"5min":0.16,"15min":0.14599999999999999},"max":{"1min":1.6519999999999999,"5min":4.3029999999999999,"15min":5.8899999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.29599999999999999,"5min":0.315,"15min":0.35999999999999999},"min":{"1min":0.22500000000000001,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":0.49199999999999999,"5min":0.60799999999999998,"15min":4.9029999999999996},"last":0.32200000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48799999999999999,"15min":0.501},"min":{"1min":0.24099999999999999,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":3.8140000000000001,"5min":4.0510000000000002,"15min":26.620999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.44900000000000001,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.19400000000000001},"max":{"1min":3.8370000000000002,"5min":3.8370000000000002,"15min":26.596},"last":0.38400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739234,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5334500,"kb_used_data":3791260,"kb_used_omap":359,"kb_used_meta":1542872,"kb_avail":89037340,"statfs":{"total":96636764160,"available":91174236160,"internally_reserved":0,"allocated":3882250240,"data_stored":4428458891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1579900962},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,4,8,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:31 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.38400000000000001,"15min":0.38700000000000001},"min":{"1min":0.20000000000000001,"5min":0.185,"15min":0.16400000000000001},"max":{"1min":0.68100000000000005,"5min":3.1339999999999999,"15min":9.218},"last":0.214},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40999999999999998,"15min":0.48899999999999999},"min":{"1min":0.22900000000000001,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":1.2829999999999999,"5min":5.6369999999999996,"15min":5.6369999999999996},"last":0.36299999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:16:31 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39500000000000002,"15min":0.45000000000000001},"min":{"1min":0.218,"5min":0.215,"15min":0.19800000000000001},"max":{"1min":0.97999999999999998,"5min":2.665,"15min":6.4119999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.41899999999999998,"15min":0.39000000000000001},"min":{"1min":0.23100000000000001,"5min":0.217,"15min":0.19800000000000001},"max":{"1min":2.3359999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4022345728,"data_stored":4594802688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3881062400,"data_stored":4427939859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7900073984,"data_stored":9019408403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:16:53.926 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:53.927 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:53.927 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:53.928 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:53.928 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:13.930 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:17:14.530 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:17:14.531 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:17:14.588 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2250,"stamp":"2023-10-21T05:17:13.703657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27860991208,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1957197,"num_read_kb":7649525,"num_write":1937839,"num_write_kb":7750335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76936,"ondisk_log_size":76936,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20382708,"kb_used_data":15494364,"kb_used_omap":1459,"kb_used_meta":4886860,"kb_avail":262732812,"statfs":{"total":289910292480,"available":269038399488,"internally_reserved":0,"allocated":15866228736,"data_stored":18103170489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":5004145204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,7,9,18,21,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17539072,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4826,"num_read_kb":19260,"num_write":4775,"num_write_kb":19100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001952"},"pg_stats":[{"pgid":"2.7","version":"84'9592","reported_seq":13237,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:09.033776+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:17:09.033776+0000","last_peered":"2023-10-21T05:17:09.033776+0000","last_clean":"2023-10-21T05:17:09.033776+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:17:09.033776+0000","last_undegraded":"2023-10-21T05:17:09.033776+0000","last_fullsized":"2023-10-21T05:17:09.033776+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9592,"log_dups_size":0,"ondisk_log_size":9592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1002,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:15.433536+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:16:15.433536+0000","last_peered":"2023-10-21T05:16:15.433536+0000","last_clean":"2023-10-21T05:16:15.433536+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:16:15.433536+0000","last_undegraded":"2023-10-21T05:16:15.433536+0000","last_fullsized":"2023-10-21T05:16:15.433536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"84'15832","reported_seq":21735,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:09.048058+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:17:09.048058+0000","last_peered":"2023-10-21T05:17:09.048058+0000","last_clean":"2023-10-21T05:17:09.048058+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:17:09.048058+0000","last_undegraded":"2023-10-21T05:17:09.048058+0000","last_fullsized":"2023-10-21T05:17:09.048058+0000","mapping_epoch":12,"log_start":"75'5800","ondisk_log_start":"75'5800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"84'3278","reported_seq":5091,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:08.707528+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:17:08.707528+0000","last_peered":"2023-10-21T05:17:08.707528+0000","last_clean":"2023-10-21T05:17:08.707528+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:17:08.707528+0000","last_undegraded":"2023-10-21T05:17:08.707528+0000","last_fullsized":"2023-10-21T05:17:08.707528+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3278,"log_dups_size":0,"ondisk_log_size":3278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"84'489971","reported_seq":976583,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:09.132296+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:17:09.132296+0000","last_peered":"2023-10-21T05:17:09.132296+0000","last_clean":"2023-10-21T05:17:09.132296+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:17:09.132296+0000","last_undegraded":"2023-10-21T05:17:09.132296+0000","last_fullsized":"2023-10-21T05:17:09.132296+0000","mapping_epoch":15,"log_start":"83'479917","ondisk_log_start":"83'479917","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7001329664,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":487935,"num_read_kb":1921408,"num_write":486213,"num_write_kb":1944852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'16092","reported_seq":22219,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:08.708182+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:17:08.708182+0000","last_peered":"2023-10-21T05:17:08.708182+0000","last_clean":"2023-10-21T05:17:08.708182+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:17:08.708182+0000","last_undegraded":"2023-10-21T05:17:08.708182+0000","last_fullsized":"2023-10-21T05:17:08.708182+0000","mapping_epoch":12,"log_start":"75'6000","ondisk_log_start":"75'6000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"84'483544","reported_seq":963140,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:13.497068+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:17:13.497068+0000","last_peered":"2023-10-21T05:17:13.497068+0000","last_clean":"2023-10-21T05:17:13.497068+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:17:13.497068+0000","last_undegraded":"2023-10-21T05:17:13.497068+0000","last_fullsized":"2023-10-21T05:17:13.497068+0000","mapping_epoch":15,"log_start":"83'473516","ondisk_log_start":"83'473516","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":33,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 7s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6908268544,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":481035,"num_read_kb":1893460,"num_write":481771,"num_write_kb":1927084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6384,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:08.708930+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:17:08.708930+0000","last_peered":"2023-10-21T05:17:08.708930+0000","last_clean":"2023-10-21T05:17:08.708930+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:17:08.708930+0000","last_undegraded":"2023-10-21T05:17:08.708930+0000","last_fullsized":"2023-10-21T05:17:08.708930+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"84'503296","reported_seq":1001344,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:09.139799+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:17:09.139799+0000","last_peered":"2023-10-21T05:17:09.139799+0000","last_clean":"2023-10-21T05:17:09.139799+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:17:09.139799+0000","last_undegraded":"2023-10-21T05:17:09.139799+0000","last_fullsized":"2023-10-21T05:17:09.139799+0000","mapping_epoch":15,"log_start":"83'493225","ondisk_log_start":"83'493225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2936,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 363s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7199256576,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":499454,"num_read_kb":1966700,"num_write":499576,"num_write_kb":1998304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":160,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:28.486629+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:15:28.486629+0000","last_peered":"2023-10-21T05:15:28.486629+0000","last_clean":"2023-10-21T05:15:28.486629+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:15:28.486629+0000","last_undegraded":"2023-10-21T05:15:28.486629+0000","last_fullsized":"2023-10-21T05:15:28.486629+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"84'473104","reported_seq":942449,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:13.486094+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:17:13.486094+0000","last_peered":"2023-10-21T05:17:13.486094+0000","last_clean":"2023-10-21T05:17:13.486094+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:17:13.486094+0000","last_undegraded":"2023-10-21T05:17:13.486094+0000","last_fullsized":"2023-10-21T05:17:13.486094+0000","mapping_epoch":15,"log_start":"83'463018","ondisk_log_start":"83'463018","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6751543315,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":470608,"num_read_kb":1852480,"num_write":469473,"num_write_kb":1877886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"84'3257","reported_seq":4605,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:08.707578+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:17:08.707578+0000","last_peered":"2023-10-21T05:17:08.707578+0000","last_clean":"2023-10-21T05:17:08.707578+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:17:08.707578+0000","last_undegraded":"2023-10-21T05:17:08.707578+0000","last_fullsized":"2023-10-21T05:17:08.707578+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":257,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:15:23.008635+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:15:23.008635+0000","last_peered":"2023-10-21T05:15:23.008635+0000","last_clean":"2023-10-21T05:15:23.008635+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:15:23.008635+0000","last_undegraded":"2023-10-21T05:15:23.008635+0000","last_fullsized":"2023-10-21T05:15:23.008635+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27860398099,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1939032,"num_read_kb":7634048,"num_write":1937033,"num_write_kb":7748126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15864762368,"data_stored":18103431206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40239,"ondisk_log_size":40239,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36598,"ondisk_log_size":36598,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739239,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5683664,"kb_used_data":3943944,"kb_used_omap":373,"kb_used_meta":1739338,"kb_avail":88688176,"statfs":{"total":96636764160,"available":90816692224,"internally_reserved":0,"allocated":4038598656,"data_stored":4610471971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1781082672},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1,8,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:17:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.38200000000000001,"15min":0.40600000000000003},"min":{"1min":0.16800000000000001,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":4.0259999999999998,"5min":4.0259999999999998,"15min":5.9009999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":4.2530000000000001,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.26500000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:17:04 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.41899999999999998,"15min":0.53200000000000003},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.17899999999999999},"max":{"1min":4.2990000000000004,"5min":4.2990000000000004,"15min":14.318},"last":0.312},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.46400000000000002,"15min":0.58099999999999996},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.20899999999999999},"max":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":14.343},"last":0.33900000000000002}]}]},{"osd":2,"up_from":8,"seq":34359739238,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9313612,"kb_used_data":7744260,"kb_used_omap":726,"kb_used_meta":1568617,"kb_avail":85058228,"statfs":{"total":96636764160,"available":87099625472,"internally_reserved":0,"allocated":7930122240,"data_stored":9048982027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":1606264802},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.41699999999999998,"15min":0.47199999999999998},"min":{"1min":0.157,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":4.1600000000000001,"5min":4.3029999999999999,"15min":5.8899999999999997},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.318,"15min":0.35399999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.46600000000000003,"5min":0.60799999999999998,"15min":4.9029999999999996},"last":0.27700000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.432,"15min":0.48999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.21199999999999999},"max":{"1min":0.51400000000000001,"5min":3.8140000000000001,"15min":26.620999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.313,"5min":0.435,"15min":0.55800000000000005},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":0.39900000000000002,"5min":3.8370000000000002,"15min":26.596},"last":0.496}]}]},{"osd":1,"up_from":8,"seq":34359739238,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5385432,"kb_used_data":3806160,"kb_used_omap":359,"kb_used_meta":1578904,"kb_avail":88986408,"statfs":{"total":96636764160,"available":91122081792,"internally_reserved":0,"allocated":3897507840,"data_stored":4443716491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1616797730},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:31 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.38400000000000001,"15min":0.38700000000000001},"min":{"1min":0.20000000000000001,"5min":0.185,"15min":0.16400000000000001},"max":{"1min":0.68100000000000005,"5min":3.1339999999999999,"15min":9.218},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40999999999999998,"15min":0.48899999999999999},"min":{"1min":0.22900000000000001,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":1.2829999999999999,"5min":5.6369999999999996,"15min":5.6369999999999996},"last":0.245}]},{"osd":2,"last update":"Sat Oct 21 05:16:31 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39500000000000002,"15min":0.45000000000000001},"min":{"1min":0.218,"5min":0.215,"15min":0.19800000000000001},"max":{"1min":0.97999999999999998,"5min":2.665,"15min":6.4119999999999999},"last":0.309},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.41899999999999998,"15min":0.39000000000000001},"min":{"1min":0.23100000000000001,"5min":0.217,"15min":0.19800000000000001},"max":{"1min":2.3359999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4037791744,"data_stored":4610248704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3896299520,"data_stored":4443176979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7930671104,"data_stored":9050005523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:17:14.589 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:14.590 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:14.590 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:15.588 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:17:15.583+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:17:31.260 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:17:31.255+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:17:31.261 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:17:31.255+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:17:31.261 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:17:31.255+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:17:31.261 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:17:31.255+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:17:31.262 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:17:31.255+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:17:34.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:17:35.145 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:17:35.145 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:17:35.172 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2261,"stamp":"2023-10-21T05:17:33.710983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27888848104,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1964957,"num_read_kb":7680397,"num_write":1945500,"num_write_kb":7780979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76947,"ondisk_log_size":76947,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20590728,"kb_used_data":15555756,"kb_used_omap":1459,"kb_used_meta":5033484,"kb_avail":262524792,"statfs":{"total":289910292480,"available":268825387008,"internally_reserved":0,"allocated":15929094144,"data_stored":18166031410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":5154288180},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,6,8,8,15,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13901824,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3824,"num_read_kb":15196,"num_write":3816,"num_write_kb":15264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001695"},"pg_stats":[{"pgid":"2.7","version":"85'9644","reported_seq":13302,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:32.307617+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:17:32.307617+0000","last_peered":"2023-10-21T05:17:32.307617+0000","last_clean":"2023-10-21T05:17:32.307617+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:17:32.307617+0000","last_undegraded":"2023-10-21T05:17:32.307617+0000","last_fullsized":"2023-10-21T05:17:32.307617+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1003,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:38.391891+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:16:38.391891+0000","last_peered":"2023-10-21T05:16:38.391891+0000","last_clean":"2023-10-21T05:16:38.391891+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:16:38.391891+0000","last_undegraded":"2023-10-21T05:16:38.391891+0000","last_fullsized":"2023-10-21T05:16:38.391891+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'15917","reported_seq":21841,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:32.328208+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:17:32.328208+0000","last_peered":"2023-10-21T05:17:32.328208+0000","last_clean":"2023-10-21T05:17:32.328208+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:17:32.328208+0000","last_undegraded":"2023-10-21T05:17:32.328208+0000","last_fullsized":"2023-10-21T05:17:32.328208+0000","mapping_epoch":12,"log_start":"75'5900","ondisk_log_start":"75'5900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"85'3296","reported_seq":5114,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:31.245560+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:17:31.245560+0000","last_peered":"2023-10-21T05:17:31.245560+0000","last_clean":"2023-10-21T05:17:31.245560+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:17:31.245560+0000","last_undegraded":"2023-10-21T05:17:31.245560+0000","last_fullsized":"2023-10-21T05:17:31.245560+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3296,"log_dups_size":0,"ondisk_log_size":3296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"84'491931","reported_seq":980485,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:29.191915+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:17:29.191915+0000","last_peered":"2023-10-21T05:17:29.191915+0000","last_clean":"2023-10-21T05:17:29.191915+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:17:29.191915+0000","last_undegraded":"2023-10-21T05:17:29.191915+0000","last_fullsized":"2023-10-21T05:17:29.191915+0000","mapping_epoch":15,"log_start":"83'481917","ondisk_log_start":"83'481917","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7008497664,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":489877,"num_read_kb":1929148,"num_write":488173,"num_write_kb":1952692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'16203","reported_seq":22350,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:29.188338+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:17:29.188338+0000","last_peered":"2023-10-21T05:17:29.188338+0000","last_clean":"2023-10-21T05:17:29.188338+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:17:29.188338+0000","last_undegraded":"2023-10-21T05:17:29.188338+0000","last_fullsized":"2023-10-21T05:17:29.188338+0000","mapping_epoch":12,"log_start":"75'6200","ondisk_log_start":"75'6200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"85'485500","reported_seq":967031,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:33.591244+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:17:33.591244+0000","last_peered":"2023-10-21T05:17:33.591244+0000","last_clean":"2023-10-21T05:17:33.591244+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:17:33.591244+0000","last_undegraded":"2023-10-21T05:17:33.591244+0000","last_fullsized":"2023-10-21T05:17:33.591244+0000","mapping_epoch":15,"log_start":"83'475417","ondisk_log_start":"83'475417","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":188,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 27s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6914957312,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":482958,"num_read_kb":1901108,"num_write":483627,"num_write_kb":1934508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6399,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:24.071909+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:17:24.071909+0000","last_peered":"2023-10-21T05:17:24.071909+0000","last_clean":"2023-10-21T05:17:24.071909+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:17:24.071909+0000","last_undegraded":"2023-10-21T05:17:24.071909+0000","last_fullsized":"2023-10-21T05:17:24.071909+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"84'505313","reported_seq":1005398,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:29.192527+0000","last_change":"2023-10-21T05:11:05.541634+0000","last_active":"2023-10-21T05:17:29.192527+0000","last_peered":"2023-10-21T05:17:29.192527+0000","last_clean":"2023-10-21T05:17:29.192527+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:17:29.192527+0000","last_undegraded":"2023-10-21T05:17:29.192527+0000","last_fullsized":"2023-10-21T05:17:29.192527+0000","mapping_epoch":15,"log_start":"83'495225","ondisk_log_start":"83'495225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":3042,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 383s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7206408192,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":501476,"num_read_kb":1974740,"num_write":501526,"num_write_kb":2006104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":161,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:15.439960+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:16:15.439960+0000","last_peered":"2023-10-21T05:16:15.439960+0000","last_clean":"2023-10-21T05:16:15.439960+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:16:15.439960+0000","last_undegraded":"2023-10-21T05:16:15.439960+0000","last_fullsized":"2023-10-21T05:16:15.439960+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"85'474999","reported_seq":946218,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:33.601403+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:17:33.601403+0000","last_peered":"2023-10-21T05:17:33.601403+0000","last_clean":"2023-10-21T05:17:33.601403+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:17:33.601403+0000","last_undegraded":"2023-10-21T05:17:33.601403+0000","last_fullsized":"2023-10-21T05:17:33.601403+0000","mapping_epoch":15,"log_start":"83'464918","ondisk_log_start":"83'464918","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6758391827,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":472481,"num_read_kb":1859924,"num_write":471368,"num_write_kb":1885466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'3275","reported_seq":4628,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:31.245788+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:17:31.245788+0000","last_peered":"2023-10-21T05:17:31.245788+0000","last_clean":"2023-10-21T05:17:31.245788+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:17:31.245788+0000","last_undegraded":"2023-10-21T05:17:31.245788+0000","last_fullsized":"2023-10-21T05:17:31.245788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3275,"log_dups_size":0,"ondisk_log_size":3275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":259,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:23.548050+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:17:23.548050+0000","last_peered":"2023-10-21T05:17:23.548050+0000","last_clean":"2023-10-21T05:17:23.548050+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:17:23.548050+0000","last_undegraded":"2023-10-21T05:17:23.548050+0000","last_fullsized":"2023-10-21T05:17:23.548050+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27888254995,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1946792,"num_read_kb":7664920,"num_write":1944694,"num_write_kb":7778770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15927566336,"data_stored":18166235174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40266,"ondisk_log_size":40266,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36582,"ondisk_log_size":36582,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739243,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5735604,"kb_used_data":3959784,"kb_used_omap":373,"kb_used_meta":1775434,"kb_avail":88636236,"statfs":{"total":96636764160,"available":90763505664,"internally_reserved":0,"allocated":4054818816,"data_stored":4626692131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1818044976},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,4,1,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:17:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.38200000000000001,"15min":0.40600000000000003},"min":{"1min":0.16800000000000001,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":4.0259999999999998,"5min":4.0259999999999998,"15min":5.9009999999999998},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":4.2530000000000001,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.24399999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:17:04 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.41899999999999998,"15min":0.53200000000000003},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.17899999999999999},"max":{"1min":4.2990000000000004,"5min":4.2990000000000004,"15min":14.318},"last":0.221},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.46400000000000002,"15min":0.58099999999999996},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.20899999999999999},"max":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":14.343},"last":0.29099999999999998}]}]},{"osd":2,"up_from":8,"seq":34359739242,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9412320,"kb_used_data":7774936,"kb_used_omap":726,"kb_used_meta":1636649,"kb_avail":84959520,"statfs":{"total":96636764160,"available":86998548480,"internally_reserved":0,"allocated":7961534464,"data_stored":9080394251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":1675929570},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.41699999999999998,"15min":0.47199999999999998},"min":{"1min":0.157,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":4.1600000000000001,"5min":4.3029999999999999,"15min":5.8899999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.318,"15min":0.35399999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.46600000000000003,"5min":0.60799999999999998,"15min":4.9029999999999996},"last":0.33900000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.432,"15min":0.48999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.21199999999999999},"max":{"1min":0.51400000000000001,"5min":3.8140000000000001,"15min":26.620999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.313,"5min":0.435,"15min":0.55800000000000005},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":0.39900000000000002,"5min":3.8370000000000002,"15min":26.596},"last":0.50700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739242,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5442804,"kb_used_data":3821036,"kb_used_omap":359,"kb_used_meta":1621400,"kb_avail":88929036,"statfs":{"total":96636764160,"available":91063332864,"internally_reserved":0,"allocated":3912740864,"data_stored":4458945028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1660313634},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,3,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:31 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.38400000000000001,"15min":0.38700000000000001},"min":{"1min":0.20000000000000001,"5min":0.185,"15min":0.16400000000000001},"max":{"1min":0.68100000000000005,"5min":3.1339999999999999,"15min":9.218},"last":0.184},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40999999999999998,"15min":0.48899999999999999},"min":{"1min":0.22900000000000001,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":1.2829999999999999,"5min":5.6369999999999996,"15min":5.6369999999999996},"last":5.6989999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:16:31 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39500000000000002,"15min":0.45000000000000001},"min":{"1min":0.218,"5min":0.215,"15min":0.19800000000000001},"max":{"1min":0.97999999999999998,"5min":2.665,"15min":6.4119999999999999},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.41899999999999998,"15min":0.39000000000000001},"min":{"1min":0.23100000000000001,"5min":0.217,"15min":0.19800000000000001},"max":{"1min":2.3359999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.221}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4053811200,"data_stored":4626268160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3911655424,"data_stored":4458532883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7962099712,"data_stored":9081434131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:17:35.174 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:35.175 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:35.175 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:40.932 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:17:40.930+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:17:40.935 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:17:40.934+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:17:40.935 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:17:40.934+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:17:40.935 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:17:40.934+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:17:40.937 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:17:40.934+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:17:40.937 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:17:40.934+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:17:40.938 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:17:40.934+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:17:55.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:17:55.698 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:17:55.699 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:17:55.730 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2271,"stamp":"2023-10-21T05:17:53.714235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27916905704,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1972398,"num_read_kb":7710001,"num_write":1953187,"num_write_kb":7811727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77005,"ondisk_log_size":77005,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20792468,"kb_used_data":15617276,"kb_used_omap":1459,"kb_used_meta":5173708,"kb_avail":262323052,"statfs":{"total":289910292480,"available":268618805248,"internally_reserved":0,"allocated":15992090624,"data_stored":18229018916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":5297877556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,10,8,12,13,19,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17928192,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4662,"num_read_kb":18536,"num_write":4899,"num_write_kb":19596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001942"},"pg_stats":[{"pgid":"2.7","version":"85'9674","reported_seq":13344,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:53.446211+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:17:53.446211+0000","last_peered":"2023-10-21T05:17:53.446211+0000","last_clean":"2023-10-21T05:17:53.446211+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:17:53.446211+0000","last_undegraded":"2023-10-21T05:17:53.446211+0000","last_fullsized":"2023-10-21T05:17:53.446211+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9674,"log_dups_size":0,"ondisk_log_size":9674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1003,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:38.391891+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:16:38.391891+0000","last_peered":"2023-10-21T05:16:38.391891+0000","last_clean":"2023-10-21T05:16:38.391891+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:16:38.391891+0000","last_undegraded":"2023-10-21T05:16:38.391891+0000","last_fullsized":"2023-10-21T05:16:38.391891+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'15967","reported_seq":21911,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:53.457124+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:17:53.457124+0000","last_peered":"2023-10-21T05:17:53.457124+0000","last_clean":"2023-10-21T05:17:53.457124+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:17:53.457124+0000","last_undegraded":"2023-10-21T05:17:53.457124+0000","last_fullsized":"2023-10-21T05:17:53.457124+0000","mapping_epoch":12,"log_start":"75'5900","ondisk_log_start":"75'5900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"85'3307","reported_seq":5129,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:49.667521+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:17:49.667521+0000","last_peered":"2023-10-21T05:17:49.667521+0000","last_clean":"2023-10-21T05:17:49.667521+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:17:49.667521+0000","last_undegraded":"2023-10-21T05:17:49.667521+0000","last_fullsized":"2023-10-21T05:17:49.667521+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'493868","reported_seq":984327,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:49.247642+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:17:49.247642+0000","last_peered":"2023-10-21T05:17:49.247642+0000","last_clean":"2023-10-21T05:17:49.247642+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:17:49.247642+0000","last_undegraded":"2023-10-21T05:17:49.247642+0000","last_fullsized":"2023-10-21T05:17:49.247642+0000","mapping_epoch":15,"log_start":"83'483817","ondisk_log_start":"83'483817","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7015624704,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":491782,"num_read_kb":1936740,"num_write":490110,"num_write_kb":1960440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"85'16273","reported_seq":22436,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:49.227325+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:17:49.227325+0000","last_peered":"2023-10-21T05:17:49.227325+0000","last_clean":"2023-10-21T05:17:49.227325+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:17:49.227325+0000","last_undegraded":"2023-10-21T05:17:49.227325+0000","last_fullsized":"2023-10-21T05:17:49.227325+0000","mapping_epoch":12,"log_start":"75'6200","ondisk_log_start":"75'6200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"85'487559","reported_seq":970949,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:53.655715+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:17:53.655715+0000","last_peered":"2023-10-21T05:17:53.655715+0000","last_clean":"2023-10-21T05:17:53.655715+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:17:53.655715+0000","last_undegraded":"2023-10-21T05:17:53.655715+0000","last_fullsized":"2023-10-21T05:17:53.655715+0000","mapping_epoch":15,"log_start":"83'477517","ondisk_log_start":"83'477517","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":374,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 47s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6922014720,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":484796,"num_read_kb":1908420,"num_write":485576,"num_write_kb":1942304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6420,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:44.548381+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:17:44.548381+0000","last_peered":"2023-10-21T05:17:44.548381+0000","last_clean":"2023-10-21T05:17:44.548381+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:17:44.548381+0000","last_undegraded":"2023-10-21T05:17:44.548381+0000","last_fullsized":"2023-10-21T05:17:44.548381+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"85'507328","reported_seq":1009378,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:49.245861+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:17:49.245861+0000","last_peered":"2023-10-21T05:17:49.245861+0000","last_clean":"2023-10-21T05:17:49.245861+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:17:49.245861+0000","last_undegraded":"2023-10-21T05:17:49.245861+0000","last_fullsized":"2023-10-21T05:17:49.245861+0000","mapping_epoch":15,"log_start":"83'497325","ondisk_log_start":"83'497325","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7213408256,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":503423,"num_read_kb":1982480,"num_write":503452,"num_write_kb":2013808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":162,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:29.092752+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:17:29.092752+0000","last_peered":"2023-10-21T05:17:29.092752+0000","last_clean":"2023-10-21T05:17:29.092752+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:17:29.092752+0000","last_undegraded":"2023-10-21T05:17:29.092752+0000","last_fullsized":"2023-10-21T05:17:29.092752+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"85'476874","reported_seq":949844,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:53.655515+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:17:53.655515+0000","last_peered":"2023-10-21T05:17:53.655515+0000","last_clean":"2023-10-21T05:17:53.655515+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:17:53.655515+0000","last_undegraded":"2023-10-21T05:17:53.655515+0000","last_fullsized":"2023-10-21T05:17:53.655515+0000","mapping_epoch":15,"log_start":"83'466818","ondisk_log_start":"83'466818","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6765264915,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":474232,"num_read_kb":1866884,"num_write":473243,"num_write_kb":1892966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'3286","reported_seq":4643,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:49.667392+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:17:49.667392+0000","last_peered":"2023-10-21T05:17:49.667392+0000","last_clean":"2023-10-21T05:17:49.667392+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:17:49.667392+0000","last_undegraded":"2023-10-21T05:17:49.667392+0000","last_fullsized":"2023-10-21T05:17:49.667392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3286,"log_dups_size":0,"ondisk_log_size":3286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":259,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:23.548050+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:17:23.548050+0000","last_peered":"2023-10-21T05:17:23.548050+0000","last_clean":"2023-10-21T05:17:23.548050+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:17:23.548050+0000","last_undegraded":"2023-10-21T05:17:23.548050+0000","last_fullsized":"2023-10-21T05:17:23.548050+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27916312595,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1954233,"num_read_kb":7694524,"num_write":1952381,"num_write_kb":7809518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15990202368,"data_stored":18228871206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40152,"ondisk_log_size":40152,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36754,"ondisk_log_size":36754,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739247,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5805184,"kb_used_data":3975224,"kb_used_omap":373,"kb_used_meta":1829578,"kb_avail":88566656,"statfs":{"total":96636764160,"available":90692255744,"internally_reserved":0,"allocated":4070629376,"data_stored":4642498204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1873488432},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:17:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.38200000000000001,"15min":0.40600000000000003},"min":{"1min":0.16800000000000001,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":4.0259999999999998,"5min":4.0259999999999998,"15min":5.9009999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":4.2530000000000001,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:17:04 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.41899999999999998,"15min":0.53200000000000003},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.17899999999999999},"max":{"1min":4.2990000000000004,"5min":4.2990000000000004,"15min":14.318},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.46400000000000002,"15min":0.58099999999999996},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.20899999999999999},"max":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":14.343},"last":0.33600000000000002}]}]},{"osd":2,"up_from":8,"seq":34359739246,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9510964,"kb_used_data":7805548,"kb_used_omap":726,"kb_used_meta":1704681,"kb_avail":84860876,"statfs":{"total":96636764160,"available":86897537024,"internally_reserved":0,"allocated":7992881152,"data_stored":9111736452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":1745594338},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.41699999999999998,"15min":0.47199999999999998},"min":{"1min":0.157,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":4.1600000000000001,"5min":4.3029999999999999,"15min":5.8899999999999997},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.318,"15min":0.35399999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.46600000000000003,"5min":0.60799999999999998,"15min":4.9029999999999996},"last":0.308}]},{"osd":1,"last update":"Sat Oct 21 05:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.432,"15min":0.48999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.21199999999999999},"max":{"1min":0.51400000000000001,"5min":3.8140000000000001,"15min":26.620999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.313,"5min":0.435,"15min":0.55800000000000005},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":0.39900000000000002,"5min":3.8370000000000002,"15min":26.596},"last":0.34799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739246,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5476320,"kb_used_data":3836504,"kb_used_omap":359,"kb_used_meta":1639448,"kb_avail":88895520,"statfs":{"total":96636764160,"available":91029012480,"internally_reserved":0,"allocated":3928580096,"data_stored":4474784260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1678794786},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,6,5,19,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.44900000000000001,"15min":0.41999999999999998},"min":{"1min":0.184,"5min":0.184,"15min":0.16400000000000001},"max":{"1min":5.0490000000000004,"5min":5.0490000000000004,"15min":9.218},"last":0.221},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.40799999999999997,"15min":0.48499999999999999},"min":{"1min":0.21299999999999999,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":5.6989999999999998,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:17:34 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.42299999999999999,"15min":0.44},"min":{"1min":0.23899999999999999,"5min":0.215,"15min":0.19800000000000001},"max":{"1min":3.5649999999999999,"5min":3.5649999999999999,"15min":6.4119999999999999},"last":0.254},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.39400000000000002,"15min":0.39100000000000001},"min":{"1min":0.221,"5min":0.217,"15min":0.19800000000000001},"max":{"1min":0.53300000000000003,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4069629952,"data_stored":4642086912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3927322624,"data_stored":4474200083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7993249792,"data_stored":9112584211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:17:55.732 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:55.733 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:18:12.915 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:18:12.910+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:18:12.927 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:18:12.926+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:18:12.928 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:18:12.926+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:18:12.928 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:18:12.926+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:18:12.928 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:18:12.926+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:18:12.929 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:18:12.926+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:18:12.929 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:18:12.926+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:18:12.929 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:18:12.926+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:18:15.734 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:18:16.177 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:18:16.178 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:18:16.208 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2282,"stamp":"2023-10-21T05:18:15.717827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27950107880,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1981310,"num_read_kb":7745485,"num_write":1962270,"num_write_kb":7848059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77001,"ondisk_log_size":77001,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21103172,"kb_used_data":15693164,"kb_used_omap":1459,"kb_used_meta":5408524,"kb_avail":262012348,"statfs":{"total":289910292480,"available":268300644352,"internally_reserved":0,"allocated":16069799936,"data_stored":18306728228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":5538329140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,1,3,9,13,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19226624,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5109,"num_read_kb":20344,"num_write":5259,"num_write_kb":21036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001956"},"pg_stats":[{"pgid":"2.7","version":"85'9716","reported_seq":13398,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:13.343231+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:18:13.343231+0000","last_peered":"2023-10-21T05:18:13.343231+0000","last_clean":"2023-10-21T05:18:13.343231+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:18:13.343231+0000","last_undegraded":"2023-10-21T05:18:13.343231+0000","last_fullsized":"2023-10-21T05:18:13.343231+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1003,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:38.391891+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:16:38.391891+0000","last_peered":"2023-10-21T05:16:38.391891+0000","last_clean":"2023-10-21T05:16:38.391891+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:16:38.391891+0000","last_undegraded":"2023-10-21T05:16:38.391891+0000","last_fullsized":"2023-10-21T05:16:38.391891+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'16037","reported_seq":22001,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:13.361800+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:18:13.361800+0000","last_peered":"2023-10-21T05:18:13.361800+0000","last_clean":"2023-10-21T05:18:13.361800+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:18:13.361800+0000","last_undegraded":"2023-10-21T05:18:13.361800+0000","last_fullsized":"2023-10-21T05:18:13.361800+0000","mapping_epoch":12,"log_start":"75'6000","ondisk_log_start":"75'6000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"85'3321","reported_seq":5147,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:12.912765+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:18:12.912765+0000","last_peered":"2023-10-21T05:18:12.912765+0000","last_clean":"2023-10-21T05:18:12.912765+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:18:12.912765+0000","last_undegraded":"2023-10-21T05:18:12.912765+0000","last_fullsized":"2023-10-21T05:18:12.912765+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3321,"log_dups_size":0,"ondisk_log_size":3321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'496419","reported_seq":989294,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:14.323908+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:18:14.323908+0000","last_peered":"2023-10-21T05:18:14.323908+0000","last_clean":"2023-10-21T05:18:14.323908+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:18:14.323908+0000","last_undegraded":"2023-10-21T05:18:14.323908+0000","last_fullsized":"2023-10-21T05:18:14.323908+0000","mapping_epoch":15,"log_start":"84'486417","ondisk_log_start":"84'486417","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7024898048,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":494197,"num_read_kb":1946376,"num_write":492661,"num_write_kb":1970644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"85'16368","reported_seq":22556,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:12.698116+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:18:12.698116+0000","last_peered":"2023-10-21T05:18:12.698116+0000","last_clean":"2023-10-21T05:18:12.698116+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:18:12.698116+0000","last_undegraded":"2023-10-21T05:18:12.698116+0000","last_fullsized":"2023-10-21T05:18:12.698116+0000","mapping_epoch":12,"log_start":"75'6300","ondisk_log_start":"75'6300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"85'489633","reported_seq":975072,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:13.735746+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:18:13.735746+0000","last_peered":"2023-10-21T05:18:13.735746+0000","last_clean":"2023-10-21T05:18:13.735746+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:18:13.735746+0000","last_undegraded":"2023-10-21T05:18:13.735746+0000","last_fullsized":"2023-10-21T05:18:13.735746+0000","mapping_epoch":15,"log_start":"84'479618","ondisk_log_start":"84'479618","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":503,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 67s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6929276928,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":486833,"num_read_kb":1916540,"num_write":487571,"num_write_kb":1950284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6446,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:10.154333+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:18:10.154333+0000","last_peered":"2023-10-21T05:18:10.154333+0000","last_clean":"2023-10-21T05:18:10.154333+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:18:10.154333+0000","last_undegraded":"2023-10-21T05:18:10.154333+0000","last_fullsized":"2023-10-21T05:18:10.154333+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"85'509856","reported_seq":1014399,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:14.323865+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:18:14.323865+0000","last_peered":"2023-10-21T05:18:14.323865+0000","last_clean":"2023-10-21T05:18:14.323865+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:18:14.323865+0000","last_undegraded":"2023-10-21T05:18:14.323865+0000","last_fullsized":"2023-10-21T05:18:14.323865+0000","mapping_epoch":15,"log_start":"84'499825","ondisk_log_start":"84'499825","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7222743040,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":505915,"num_read_kb":1992380,"num_write":505980,"num_write_kb":2023920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":162,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:29.092752+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:17:29.092752+0000","last_peered":"2023-10-21T05:17:29.092752+0000","last_clean":"2023-10-21T05:17:29.092752+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:17:29.092752+0000","last_undegraded":"2023-10-21T05:17:29.092752+0000","last_fullsized":"2023-10-21T05:17:29.092752+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"85'478883","reported_seq":953817,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:13.740306+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:18:13.740306+0000","last_peered":"2023-10-21T05:18:13.740306+0000","last_clean":"2023-10-21T05:18:13.740306+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:18:13.740306+0000","last_undegraded":"2023-10-21T05:18:13.740306+0000","last_fullsized":"2023-10-21T05:18:13.740306+0000","mapping_epoch":15,"log_start":"84'468818","ondisk_log_start":"84'468818","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6772596755,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":476200,"num_read_kb":1874712,"num_write":475252,"num_write_kb":1901002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'3300","reported_seq":4661,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:12.912255+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:18:12.912255+0000","last_peered":"2023-10-21T05:18:12.912255+0000","last_clean":"2023-10-21T05:18:12.912255+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:18:12.912255+0000","last_undegraded":"2023-10-21T05:18:12.912255+0000","last_fullsized":"2023-10-21T05:18:12.912255+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3300,"log_dups_size":0,"ondisk_log_size":3300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":259,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:23.548050+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:17:23.548050+0000","last_peered":"2023-10-21T05:17:23.548050+0000","last_clean":"2023-10-21T05:17:23.548050+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:17:23.548050+0000","last_undegraded":"2023-10-21T05:17:23.548050+0000","last_fullsized":"2023-10-21T05:17:23.548050+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27949514771,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1963145,"num_read_kb":7730008,"num_write":1961464,"num_write_kb":7845850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16068624384,"data_stored":18307293222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40113,"ondisk_log_size":40113,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36789,"ondisk_log_size":36789,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739252,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5865676,"kb_used_data":3995460,"kb_used_omap":373,"kb_used_meta":1869834,"kb_avail":88506164,"statfs":{"total":96636764160,"available":90630311936,"internally_reserved":0,"allocated":4091351040,"data_stored":4663219868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1914710576},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:18:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.44400000000000001,"15min":0.40799999999999997},"min":{"1min":0.189,"5min":0.16800000000000001,"15min":0.14299999999999999},"max":{"1min":3.5329999999999999,"5min":4.0259999999999998,"15min":4.9889999999999999},"last":0.17799999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.51600000000000001,"15min":0.45200000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.3279999999999998,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.246}]},{"osd":2,"last update":"Sat Oct 21 05:18:04 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.52700000000000002,"15min":0.55400000000000005},"min":{"1min":0.221,"5min":0.19900000000000001,"15min":0.17899999999999999},"max":{"1min":5.4390000000000001,"5min":5.4390000000000001,"15min":14.318},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.57199999999999995,"15min":0.60199999999999998},"min":{"1min":0.28100000000000003,"5min":0.26200000000000001,"15min":0.20899999999999999},"max":{"1min":5.4740000000000002,"5min":5.4740000000000002,"15min":14.343},"last":0.42199999999999999}]}]},{"osd":2,"up_from":8,"seq":34359739251,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9655060,"kb_used_data":7845260,"kb_used_omap":726,"kb_used_meta":1809065,"kb_avail":84716780,"statfs":{"total":96636764160,"available":86749982720,"internally_reserved":0,"allocated":8033546240,"data_stored":9152401540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":1852483554},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:58 2023","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.40999999999999998,"15min":0.44600000000000001},"min":{"1min":0.17499999999999999,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":0.46300000000000002,"5min":4.3029999999999999,"15min":5.8899999999999997},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.307,"5min":0.316,"15min":0.34699999999999998},"min":{"1min":0.20699999999999999,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.57299999999999995,"5min":0.59499999999999997,"15min":4.9029999999999996},"last":0.40100000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:17:58 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.438,"15min":0.497},"min":{"1min":0.28000000000000003,"5min":0.23200000000000001,"15min":0.21199999999999999},"max":{"1min":0.56100000000000005,"5min":3.8140000000000001,"15min":26.620999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.437,"15min":0.53800000000000003},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":0.61199999999999999,"5min":3.8370000000000002,"15min":26.596},"last":0.436}]}]},{"osd":1,"up_from":8,"seq":34359739250,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5582436,"kb_used_data":3852444,"kb_used_omap":359,"kb_used_meta":1729624,"kb_avail":88789404,"statfs":{"total":96636764160,"available":90920349696,"internally_reserved":0,"allocated":3944902656,"data_stored":4491106820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1771135010},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,1,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.44900000000000001,"15min":0.41999999999999998},"min":{"1min":0.184,"5min":0.184,"15min":0.16400000000000001},"max":{"1min":5.0490000000000004,"5min":5.0490000000000004,"15min":9.218},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.40799999999999997,"15min":0.48499999999999999},"min":{"1min":0.21299999999999999,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":5.6989999999999998,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.39600000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:17:34 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.42299999999999999,"15min":0.44},"min":{"1min":0.23899999999999999,"5min":0.215,"15min":0.19800000000000001},"max":{"1min":3.5649999999999999,"5min":3.5649999999999999,"15min":6.4119999999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.39400000000000002,"15min":0.39100000000000001},"min":{"1min":0.221,"5min":0.217,"15min":0.19800000000000001},"max":{"1min":0.53300000000000003,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.378}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4090441728,"data_stored":4662898688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3943723008,"data_stored":4490600467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8034459648,"data_stored":9153794067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:18:16.210 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:16.211 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:18:36.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:18:36.825 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:18:36.826 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:18:36.857 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2292,"stamp":"2023-10-21T05:18:35.721007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27979390184,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1988970,"num_read_kb":7775997,"num_write":1970289,"num_write_kb":7880135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77022,"ondisk_log_size":77022,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21299484,"kb_used_data":15757572,"kb_used_omap":1459,"kb_used_meta":5540428,"kb_avail":261816036,"statfs":{"total":289910292480,"available":268099620864,"internally_reserved":0,"allocated":16135753728,"data_stored":18372682020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":5673398836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,4,10,18,10,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21917696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5815,"num_read_kb":23188,"num_write":6014,"num_write_kb":24056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001847"},"pg_stats":[{"pgid":"2.7","version":"85'9758","reported_seq":13452,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:33.139932+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:18:33.139932+0000","last_peered":"2023-10-21T05:18:33.139932+0000","last_clean":"2023-10-21T05:18:33.139932+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:18:33.139932+0000","last_undegraded":"2023-10-21T05:18:33.139932+0000","last_fullsized":"2023-10-21T05:18:33.139932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9758,"log_dups_size":0,"ondisk_log_size":9758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1003,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:16:38.391891+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:16:38.391891+0000","last_peered":"2023-10-21T05:16:38.391891+0000","last_clean":"2023-10-21T05:16:38.391891+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:16:38.391891+0000","last_undegraded":"2023-10-21T05:16:38.391891+0000","last_fullsized":"2023-10-21T05:16:38.391891+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'16107","reported_seq":22092,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:33.159427+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:18:33.159427+0000","last_peered":"2023-10-21T05:18:33.159427+0000","last_clean":"2023-10-21T05:18:33.159427+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:18:33.159427+0000","last_undegraded":"2023-10-21T05:18:33.159427+0000","last_fullsized":"2023-10-21T05:18:33.159427+0000","mapping_epoch":12,"log_start":"75'6100","ondisk_log_start":"75'6100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"85'3335","reported_seq":5166,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:33.011047+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:18:33.011047+0000","last_peered":"2023-10-21T05:18:33.011047+0000","last_clean":"2023-10-21T05:18:33.011047+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:18:33.011047+0000","last_undegraded":"2023-10-21T05:18:33.011047+0000","last_fullsized":"2023-10-21T05:18:33.011047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3335,"log_dups_size":0,"ondisk_log_size":3335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'498443","reported_seq":993220,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:34.357221+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:18:34.357221+0000","last_peered":"2023-10-21T05:18:34.357221+0000","last_clean":"2023-10-21T05:18:34.357221+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:18:34.357221+0000","last_undegraded":"2023-10-21T05:18:34.357221+0000","last_fullsized":"2023-10-21T05:18:34.357221+0000","mapping_epoch":15,"log_start":"84'488417","ondisk_log_start":"84'488417","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7032262656,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":496102,"num_read_kb":1953952,"num_write":494685,"num_write_kb":1978740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"85'16443","reported_seq":22652,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:34.248369+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:18:34.248369+0000","last_peered":"2023-10-21T05:18:34.248369+0000","last_clean":"2023-10-21T05:18:34.248369+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:18:34.248369+0000","last_undegraded":"2023-10-21T05:18:34.248369+0000","last_fullsized":"2023-10-21T05:18:34.248369+0000","mapping_epoch":12,"log_start":"75'6400","ondisk_log_start":"75'6400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"85'491753","reported_seq":979037,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:33.848216+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:18:33.848216+0000","last_peered":"2023-10-21T05:18:33.848216+0000","last_clean":"2023-10-21T05:18:33.848216+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:18:33.848216+0000","last_undegraded":"2023-10-21T05:18:33.848216+0000","last_fullsized":"2023-10-21T05:18:33.848216+0000","mapping_epoch":15,"log_start":"84'481718","ondisk_log_start":"84'481718","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":646,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 87s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6936625152,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":488660,"num_read_kb":1923832,"num_write":489604,"num_write_kb":1958416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6466,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:30.632536+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:18:30.632536+0000","last_peered":"2023-10-21T05:18:30.632536+0000","last_clean":"2023-10-21T05:18:30.632536+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:18:30.632536+0000","last_undegraded":"2023-10-21T05:18:30.632536+0000","last_fullsized":"2023-10-21T05:18:30.632536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"85'511931","reported_seq":1018516,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:34.354587+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:18:34.354587+0000","last_peered":"2023-10-21T05:18:34.354587+0000","last_clean":"2023-10-21T05:18:34.354587+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:18:34.354587+0000","last_undegraded":"2023-10-21T05:18:34.354587+0000","last_fullsized":"2023-10-21T05:18:34.354587+0000","mapping_epoch":15,"log_start":"84'501925","ondisk_log_start":"84'501925","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7230484480,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":507958,"num_read_kb":2000516,"num_write":508055,"num_write_kb":2032220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":162,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:29.092752+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:17:29.092752+0000","last_peered":"2023-10-21T05:17:29.092752+0000","last_clean":"2023-10-21T05:17:29.092752+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:17:29.092752+0000","last_undegraded":"2023-10-21T05:17:29.092752+0000","last_fullsized":"2023-10-21T05:17:29.092752+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"85'480770","reported_seq":957592,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:33.876468+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:18:33.876468+0000","last_peered":"2023-10-21T05:18:33.876468+0000","last_clean":"2023-10-21T05:18:33.876468+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:18:33.876468+0000","last_undegraded":"2023-10-21T05:18:33.876468+0000","last_fullsized":"2023-10-21T05:18:33.876468+0000","mapping_epoch":15,"log_start":"84'470718","ondisk_log_start":"84'470718","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6779424787,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":478085,"num_read_kb":1882220,"num_write":477139,"num_write_kb":1908550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'3314","reported_seq":4680,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:18:33.022120+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:18:33.022120+0000","last_peered":"2023-10-21T05:18:33.022120+0000","last_clean":"2023-10-21T05:18:33.022120+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:18:33.022120+0000","last_undegraded":"2023-10-21T05:18:33.022120+0000","last_fullsized":"2023-10-21T05:18:33.022120+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3314,"log_dups_size":0,"ondisk_log_size":3314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":259,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:17:23.548050+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:17:23.548050+0000","last_peered":"2023-10-21T05:17:23.548050+0000","last_clean":"2023-10-21T05:17:23.548050+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:17:23.548050+0000","last_undegraded":"2023-10-21T05:17:23.548050+0000","last_fullsized":"2023-10-21T05:17:23.548050+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27978797075,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1970805,"num_read_kb":7760520,"num_write":1969483,"num_write_kb":7877926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16134283264,"data_stored":18372952102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40119,"ondisk_log_size":40119,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36804,"ondisk_log_size":36804,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739256,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5918340,"kb_used_data":4012028,"kb_used_omap":373,"kb_used_meta":1905930,"kb_avail":88453500,"statfs":{"total":96636764160,"available":90576384000,"internally_reserved":0,"allocated":4108316672,"data_stored":4680185500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1951672880},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:18:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.44400000000000001,"15min":0.40799999999999997},"min":{"1min":0.189,"5min":0.16800000000000001,"15min":0.14299999999999999},"max":{"1min":3.5329999999999999,"5min":4.0259999999999998,"15min":4.9889999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.51600000000000001,"15min":0.45200000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.3279999999999998,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.28499999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:18:04 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.52700000000000002,"15min":0.55400000000000005},"min":{"1min":0.221,"5min":0.19900000000000001,"15min":0.17899999999999999},"max":{"1min":5.4390000000000001,"5min":5.4390000000000001,"15min":14.318},"last":0.314},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.57199999999999995,"15min":0.60199999999999998},"min":{"1min":0.28100000000000003,"5min":0.26200000000000001,"15min":0.20899999999999999},"max":{"1min":5.4740000000000002,"5min":5.4740000000000002,"15min":14.343},"last":0.33900000000000002}]}]},{"osd":2,"up_from":8,"seq":34359739255,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9765028,"kb_used_data":7877468,"kb_used_omap":726,"kb_used_meta":1886825,"kb_avail":84606812,"statfs":{"total":96636764160,"available":86637375488,"internally_reserved":0,"allocated":8066527232,"data_stored":9185382532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":1932109794},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:58 2023","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.40999999999999998,"15min":0.44600000000000001},"min":{"1min":0.17499999999999999,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":0.46300000000000002,"5min":4.3029999999999999,"15min":5.8899999999999997},"last":7.234},{"interface":"front","average":{"1min":0.307,"5min":0.316,"15min":0.34699999999999998},"min":{"1min":0.20699999999999999,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.57299999999999995,"5min":0.59499999999999997,"15min":4.9029999999999996},"last":0.27500000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:17:58 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.438,"15min":0.497},"min":{"1min":0.28000000000000003,"5min":0.23200000000000001,"15min":0.21199999999999999},"max":{"1min":0.56100000000000005,"5min":3.8140000000000001,"15min":26.620999999999999},"last":0.372},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.437,"15min":0.53800000000000003},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":0.61199999999999999,"5min":3.8370000000000002,"15min":26.596},"last":0.307}]}]},{"osd":1,"up_from":8,"seq":34359739254,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5616116,"kb_used_data":3868076,"kb_used_omap":359,"kb_used_meta":1747672,"kb_avail":88755724,"statfs":{"total":96636764160,"available":90885861376,"internally_reserved":0,"allocated":3960909824,"data_stored":4507113988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1789616162},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,4,11,6,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:34 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.44900000000000001,"15min":0.41999999999999998},"min":{"1min":0.184,"5min":0.184,"15min":0.16400000000000001},"max":{"1min":5.0490000000000004,"5min":5.0490000000000004,"15min":9.218},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.40799999999999997,"15min":0.48499999999999999},"min":{"1min":0.21299999999999999,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":5.6989999999999998,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:17:34 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.42299999999999999,"15min":0.44},"min":{"1min":0.23899999999999999,"5min":0.215,"15min":0.19800000000000001},"max":{"1min":3.5649999999999999,"5min":3.5649999999999999,"15min":6.4119999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.39400000000000002,"15min":0.39100000000000001},"min":{"1min":0.221,"5min":0.217,"15min":0.19800000000000001},"max":{"1min":0.53300000000000003,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4107231232,"data_stored":4679688192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3959783424,"data_stored":4506660883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8067268608,"data_stored":9186603027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:18:36.859 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:36.860 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:18:56.861 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:18:57.415 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:18:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:18:57.449 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2303,"stamp":"2023-10-21T05:18:55.724454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28009864424,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1996757,"num_read_kb":7807097,"num_write":1978637,"num_write_kb":7913527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77206,"ondisk_log_size":77206,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21167708,"kb_used_data":15824324,"kb_used_omap":1459,"kb_used_meta":5341900,"kb_avail":261947812,"statfs":{"total":289910292480,"available":268234559488,"internally_reserved":0,"allocated":16204107776,"data_stored":18441022478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":5470106164},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,11,10,11,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22900736,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5833,"num_read_kb":23304,"num_write":6276,"num_write_kb":25104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002040"},"pg_stats":[{"pgid":"2.7","version":"86'9776","reported_seq":13483,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:51.109096+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:18:51.109096+0000","last_peered":"2023-10-21T05:18:51.109096+0000","last_clean":"2023-10-21T05:18:51.109096+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:18:51.109096+0000","last_undegraded":"2023-10-21T05:18:51.109096+0000","last_fullsized":"2023-10-21T05:18:51.109096+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9776,"log_dups_size":0,"ondisk_log_size":9776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1005,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:40.494463+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:18:40.494463+0000","last_peered":"2023-10-21T05:18:40.494463+0000","last_clean":"2023-10-21T05:18:40.494463+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:18:40.494463+0000","last_undegraded":"2023-10-21T05:18:40.494463+0000","last_fullsized":"2023-10-21T05:18:40.494463+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'16137","reported_seq":22143,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:51.107910+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:18:51.107910+0000","last_peered":"2023-10-21T05:18:51.107910+0000","last_clean":"2023-10-21T05:18:51.107910+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:18:51.107910+0000","last_undegraded":"2023-10-21T05:18:51.107910+0000","last_fullsized":"2023-10-21T05:18:51.107910+0000","mapping_epoch":12,"log_start":"75'6100","ondisk_log_start":"75'6100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'3341","reported_seq":5177,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:48.127280+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:18:48.127280+0000","last_peered":"2023-10-21T05:18:48.127280+0000","last_clean":"2023-10-21T05:18:48.127280+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:18:48.127280+0000","last_undegraded":"2023-10-21T05:18:48.127280+0000","last_fullsized":"2023-10-21T05:18:48.127280+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3341,"log_dups_size":0,"ondisk_log_size":3341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"86'500597","reported_seq":997366,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:54.524925+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:18:54.524925+0000","last_peered":"2023-10-21T05:18:54.524925+0000","last_clean":"2023-10-21T05:18:54.524925+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:18:54.524925+0000","last_undegraded":"2023-10-21T05:18:54.524925+0000","last_fullsized":"2023-10-21T05:18:54.524925+0000","mapping_epoch":15,"log_start":"84'490517","ondisk_log_start":"84'490517","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7040131072,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":498091,"num_read_kb":1961892,"num_write":496839,"num_write_kb":1987356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'16473","reported_seq":22703,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:51.149396+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:18:51.149396+0000","last_peered":"2023-10-21T05:18:51.149396+0000","last_clean":"2023-10-21T05:18:51.149396+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:18:51.149396+0000","last_undegraded":"2023-10-21T05:18:51.149396+0000","last_fullsized":"2023-10-21T05:18:51.149396+0000","mapping_epoch":12,"log_start":"75'6400","ondisk_log_start":"75'6400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"86'493793","reported_seq":982950,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:53.985938+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:18:53.985938+0000","last_peered":"2023-10-21T05:18:53.985938+0000","last_clean":"2023-10-21T05:18:53.985938+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:18:53.985938+0000","last_undegraded":"2023-10-21T05:18:53.985938+0000","last_fullsized":"2023-10-21T05:18:53.985938+0000","mapping_epoch":15,"log_start":"84'483718","ondisk_log_start":"84'483718","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":717,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 107s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6943891456,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":490523,"num_read_kb":1931276,"num_write":491598,"num_write_kb":1966392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"79'293","reported_seq":6487,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:51.125735+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:18:51.125735+0000","last_peered":"2023-10-21T05:18:51.125735+0000","last_clean":"2023-10-21T05:18:51.125735+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:18:51.125735+0000","last_undegraded":"2023-10-21T05:18:51.125735+0000","last_fullsized":"2023-10-21T05:18:51.125735+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"86'514165","reported_seq":1022795,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:54.510993+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:18:54.510993+0000","last_peered":"2023-10-21T05:18:54.510993+0000","last_clean":"2023-10-21T05:18:54.510993+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:18:54.510993+0000","last_undegraded":"2023-10-21T05:18:54.510993+0000","last_fullsized":"2023-10-21T05:18:54.510993+0000","mapping_epoch":15,"log_start":"84'504125","ondisk_log_start":"84'504125","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7238742016,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":510004,"num_read_kb":2008684,"num_write":510289,"num_write_kb":2041156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":163,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:30.483139+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:17:30.483139+0000","last_peered":"2023-10-21T05:17:30.483139+0000","last_clean":"2023-10-21T05:17:30.483139+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:17:30.483139+0000","last_undegraded":"2023-10-21T05:17:30.483139+0000","last_fullsized":"2023-10-21T05:17:30.483139+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"86'482736","reported_seq":961448,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:53.983039+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:18:53.983039+0000","last_peered":"2023-10-21T05:18:53.983039+0000","last_clean":"2023-10-21T05:18:53.983039+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:18:53.983039+0000","last_undegraded":"2023-10-21T05:18:53.983039+0000","last_fullsized":"2023-10-21T05:18:53.983039+0000","mapping_epoch":15,"log_start":"84'472718","ondisk_log_start":"84'472718","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6786506771,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":479974,"num_read_kb":1889768,"num_write":479105,"num_write_kb":1916414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"86'3320","reported_seq":4691,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:48.158904+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:18:48.158904+0000","last_peered":"2023-10-21T05:18:48.158904+0000","last_clean":"2023-10-21T05:18:48.158904+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:18:48.158904+0000","last_undegraded":"2023-10-21T05:18:48.158904+0000","last_fullsized":"2023-10-21T05:18:48.158904+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3320,"log_dups_size":0,"ondisk_log_size":3320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":260,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:30.476341+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:17:30.476341+0000","last_peered":"2023-10-21T05:17:30.476341+0000","last_clean":"2023-10-21T05:17:30.476341+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:17:30.476341+0000","last_undegraded":"2023-10-21T05:17:30.476341+0000","last_fullsized":"2023-10-21T05:17:30.476341+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28009271315,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1978592,"num_read_kb":7791620,"num_write":1977831,"num_write_kb":7911318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16202637312,"data_stored":18441306150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40213,"ondisk_log_size":40213,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36894,"ondisk_log_size":36894,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739260,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5633072,"kb_used_data":4029352,"kb_used_omap":373,"kb_used_meta":1603338,"kb_avail":88738768,"statfs":{"total":96636764160,"available":90868498432,"internally_reserved":0,"allocated":4126056448,"data_stored":4697920746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1641818672},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,3,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:18:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.44400000000000001,"15min":0.40799999999999997},"min":{"1min":0.189,"5min":0.16800000000000001,"15min":0.14299999999999999},"max":{"1min":3.5329999999999999,"5min":4.0259999999999998,"15min":4.9889999999999999},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.51600000000000001,"15min":0.45200000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.3279999999999998,"5min":6.2990000000000004,"15min":6.2990000000000004},"last":0.251}]},{"osd":2,"last update":"Sat Oct 21 05:18:04 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.52700000000000002,"15min":0.55400000000000005},"min":{"1min":0.221,"5min":0.19900000000000001,"15min":0.17899999999999999},"max":{"1min":5.4390000000000001,"5min":5.4390000000000001,"15min":14.318},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.57199999999999995,"15min":0.60199999999999998},"min":{"1min":0.28100000000000003,"5min":0.26200000000000001,"15min":0.20899999999999999},"max":{"1min":5.4740000000000002,"5min":5.4740000000000002,"15min":14.343},"last":0.34799999999999998}]}]},{"osd":2,"up_from":8,"seq":34359739259,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9866480,"kb_used_data":7910888,"kb_used_omap":726,"kb_used_meta":1954857,"kb_avail":84505360,"statfs":{"total":96636764160,"available":86533488640,"internally_reserved":0,"allocated":8100749312,"data_stored":9219600082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":2001774562},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:58 2023","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.40999999999999998,"15min":0.44600000000000001},"min":{"1min":0.17499999999999999,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":0.46300000000000002,"5min":4.3029999999999999,"15min":5.8899999999999997},"last":0.192},{"interface":"front","average":{"1min":0.307,"5min":0.316,"15min":0.34699999999999998},"min":{"1min":0.20699999999999999,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.57299999999999995,"5min":0.59499999999999997,"15min":4.9029999999999996},"last":0.23400000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:17:58 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.438,"15min":0.497},"min":{"1min":0.28000000000000003,"5min":0.23200000000000001,"15min":0.21199999999999999},"max":{"1min":0.56100000000000005,"5min":3.8140000000000001,"15min":26.620999999999999},"last":0.31},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.437,"15min":0.53800000000000003},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.19400000000000001},"max":{"1min":0.61199999999999999,"5min":3.8370000000000002,"15min":26.596},"last":0.29399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739258,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5668156,"kb_used_data":3884084,"kb_used_omap":359,"kb_used_meta":1783704,"kb_avail":88703684,"statfs":{"total":96636764160,"available":90832572416,"internally_reserved":0,"allocated":3977302016,"data_stored":4523501650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1826512930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,4,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.45100000000000001,"15min":0.41799999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16400000000000001},"max":{"1min":1.022,"5min":5.0490000000000004,"15min":9.218},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.377,"15min":0.46500000000000002},"min":{"1min":0.20799999999999999,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":0.623,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.21299999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.41699999999999998,"15min":0.432},"min":{"1min":0.23200000000000001,"5min":0.215,"15min":0.19800000000000001},"max":{"1min":0.68899999999999995,"5min":3.5649999999999999,"15min":6.4119999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":0.65100000000000002,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.26300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4125208576,"data_stored":4697665536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3975995392,"data_stored":4522872851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8101433344,"data_stored":9220767763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:18:57.451 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:57.452 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:19:17.453 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:19:17.932 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:19:17.933 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:19:17.964 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2314,"stamp":"2023-10-21T05:19:17.728105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28038806760,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2004577,"num_read_kb":7838301,"num_write":1986621,"num_write_kb":7945463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77247,"ondisk_log_size":77247,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21441324,"kb_used_data":15888532,"kb_used_omap":1459,"kb_used_meta":5551308,"kb_avail":261674196,"statfs":{"total":289910292480,"available":267954376704,"internally_reserved":0,"allocated":16269856768,"data_stored":18506771470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1494476,"internal_metadata":5684539956},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,4,2,10,2,2,6,21,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14151680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3795,"num_read_kb":15156,"num_write":3940,"num_write_kb":15760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002048"},"pg_stats":[{"pgid":"2.7","version":"86'9809","reported_seq":13529,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:13.702131+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:19:13.702131+0000","last_peered":"2023-10-21T05:19:13.702131+0000","last_clean":"2023-10-21T05:19:13.702131+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:19:13.702131+0000","last_undegraded":"2023-10-21T05:19:13.702131+0000","last_fullsized":"2023-10-21T05:19:13.702131+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9809,"log_dups_size":0,"ondisk_log_size":9809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1005,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:40.494463+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:18:40.494463+0000","last_peered":"2023-10-21T05:18:40.494463+0000","last_clean":"2023-10-21T05:18:40.494463+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:18:40.494463+0000","last_undegraded":"2023-10-21T05:18:40.494463+0000","last_fullsized":"2023-10-21T05:18:40.494463+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'16192","reported_seq":22216,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:13.720949+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:19:13.720949+0000","last_peered":"2023-10-21T05:19:13.720949+0000","last_clean":"2023-10-21T05:19:13.720949+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:19:13.720949+0000","last_undegraded":"2023-10-21T05:19:13.720949+0000","last_fullsized":"2023-10-21T05:19:13.720949+0000","mapping_epoch":12,"log_start":"75'6100","ondisk_log_start":"75'6100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'3352","reported_seq":5192,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:11.757197+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:19:11.757197+0000","last_peered":"2023-10-21T05:19:11.757197+0000","last_clean":"2023-10-21T05:19:11.757197+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:19:11.757197+0000","last_undegraded":"2023-10-21T05:19:11.757197+0000","last_fullsized":"2023-10-21T05:19:11.757197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3352,"log_dups_size":0,"ondisk_log_size":3352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"86'502602","reported_seq":1001310,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:14.635773+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:19:14.635773+0000","last_peered":"2023-10-21T05:19:14.635773+0000","last_clean":"2023-10-21T05:19:14.635773+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:19:14.635773+0000","last_undegraded":"2023-10-21T05:19:14.635773+0000","last_fullsized":"2023-10-21T05:19:14.635773+0000","mapping_epoch":15,"log_start":"85'492517","ondisk_log_start":"85'492517","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7047389184,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":500028,"num_read_kb":1969624,"num_write":498844,"num_write_kb":1995376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'16523","reported_seq":22773,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:12.669649+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:19:12.669649+0000","last_peered":"2023-10-21T05:19:12.669649+0000","last_clean":"2023-10-21T05:19:12.669649+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:19:12.669649+0000","last_undegraded":"2023-10-21T05:19:12.669649+0000","last_fullsized":"2023-10-21T05:19:12.669649+0000","mapping_epoch":12,"log_start":"75'6500","ondisk_log_start":"75'6500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"86'495837","reported_seq":986972,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:14.027896+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:19:14.027896+0000","last_peered":"2023-10-21T05:19:14.027896+0000","last_clean":"2023-10-21T05:19:14.027896+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:19:14.027896+0000","last_undegraded":"2023-10-21T05:19:14.027896+0000","last_fullsized":"2023-10-21T05:19:14.027896+0000","mapping_epoch":15,"log_start":"85'485819","ondisk_log_start":"85'485819","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":871,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 127s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6950973440,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":492490,"num_read_kb":1939128,"num_write":493549,"num_write_kb":1974196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6507,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:11.588274+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:19:11.588274+0000","last_peered":"2023-10-21T05:19:11.588274+0000","last_clean":"2023-10-21T05:19:11.588274+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:19:11.588274+0000","last_undegraded":"2023-10-21T05:19:11.588274+0000","last_fullsized":"2023-10-21T05:19:11.588274+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"86'516226","reported_seq":1026905,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:14.632893+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:19:14.632893+0000","last_peered":"2023-10-21T05:19:14.632893+0000","last_clean":"2023-10-21T05:19:14.632893+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:19:14.632893+0000","last_undegraded":"2023-10-21T05:19:14.632893+0000","last_fullsized":"2023-10-21T05:19:14.632893+0000","mapping_epoch":15,"log_start":"85'506225","ondisk_log_start":"85'506225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7246270464,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":512058,"num_read_kb":2016864,"num_write":512350,"num_write_kb":2049400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":163,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:30.483139+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:17:30.483139+0000","last_peered":"2023-10-21T05:17:30.483139+0000","last_clean":"2023-10-21T05:17:30.483139+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:17:30.483139+0000","last_undegraded":"2023-10-21T05:17:30.483139+0000","last_fullsized":"2023-10-21T05:17:30.483139+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"86'484703","reported_seq":965279,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:14.027240+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:19:14.027240+0000","last_peered":"2023-10-21T05:19:14.027240+0000","last_clean":"2023-10-21T05:19:14.027240+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:19:14.027240+0000","last_undegraded":"2023-10-21T05:19:14.027240+0000","last_fullsized":"2023-10-21T05:19:14.027240+0000","mapping_epoch":15,"log_start":"84'474618","ondisk_log_start":"84'474618","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6793580563,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":481836,"num_read_kb":1897208,"num_write":481072,"num_write_kb":1924282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"86'3331","reported_seq":4706,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:11.760612+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:19:11.760612+0000","last_peered":"2023-10-21T05:19:11.760612+0000","last_clean":"2023-10-21T05:19:11.760612+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:19:11.760612+0000","last_undegraded":"2023-10-21T05:19:11.760612+0000","last_fullsized":"2023-10-21T05:19:11.760612+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3331,"log_dups_size":0,"ondisk_log_size":3331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":260,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:17:30.476341+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:17:30.476341+0000","last_peered":"2023-10-21T05:17:30.476341+0000","last_clean":"2023-10-21T05:17:30.476341+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:17:30.476341+0000","last_undegraded":"2023-10-21T05:17:30.476341+0000","last_fullsized":"2023-10-21T05:17:30.476341+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28038213651,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1986412,"num_read_kb":7822824,"num_write":1985815,"num_write_kb":7943254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16267984896,"data_stored":18506653734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40189,"ondisk_log_size":40189,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36959,"ondisk_log_size":36959,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739264,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5710020,"kb_used_data":4045564,"kb_used_omap":373,"kb_used_meta":1664074,"kb_avail":88661820,"statfs":{"total":96636764160,"available":90789703680,"internally_reserved":0,"allocated":4142657536,"data_stored":4714521834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1704012336},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,2,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:19:05 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45500000000000002,"15min":0.40699999999999997},"min":{"1min":0.17799999999999999,"5min":0.16800000000000001,"15min":0.14299999999999999},"max":{"1min":2.6030000000000002,"5min":4.0259999999999998,"15min":4.9889999999999999},"last":0.188},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.54500000000000004,"15min":0.47899999999999998},"min":{"1min":0.246,"5min":0.182,"15min":0.182},"max":{"1min":6.992,"5min":6.992,"15min":6.992},"last":0.26100000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:19:05 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.17899999999999999},"max":{"1min":2.5019999999999998,"5min":5.4390000000000001,"15min":14.318},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.20899999999999999},"max":{"1min":2.5339999999999998,"5min":5.4740000000000002,"15min":14.343},"last":0.28999999999999998}]}]},{"osd":2,"up_from":8,"seq":34359739263,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9993196,"kb_used_data":7943076,"kb_used_omap":726,"kb_used_meta":2049385,"kb_avail":84378644,"statfs":{"total":96636764160,"available":86403731456,"internally_reserved":0,"allocated":8133709824,"data_stored":9252560594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":743454,"internal_metadata":2098571234},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:59 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.16900000000000001,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":7.234,"5min":7.234,"15min":7.234},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.33600000000000002,"15min":0.35799999999999998},"min":{"1min":0.217,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":4.9029999999999996},"last":0.31}]},{"osd":1,"last update":"Sat Oct 21 05:18:59 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.39900000000000002,"15min":0.497},"min":{"1min":0.249,"5min":0.23200000000000001,"15min":0.21199999999999999},"max":{"1min":0.38200000000000001,"5min":3.8140000000000001,"15min":26.620999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.41299999999999998,"15min":0.54400000000000004},"min":{"1min":0.27500000000000002,"5min":0.23000000000000001,"15min":0.20000000000000001},"max":{"1min":1.2250000000000001,"5min":3.8370000000000002,"15min":26.596},"last":0.35599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739262,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5738108,"kb_used_data":3899892,"kb_used_omap":359,"kb_used_meta":1837848,"kb_avail":88633732,"statfs":{"total":96636764160,"available":90760941568,"internally_reserved":0,"allocated":3993489408,"data_stored":4539689042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1881956386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,5,1,2,6,21,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.45100000000000001,"15min":0.41799999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16400000000000001},"max":{"1min":1.022,"5min":5.0490000000000004,"15min":9.218},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.377,"15min":0.46500000000000002},"min":{"1min":0.20799999999999999,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":0.623,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.313}]},{"osd":2,"last update":"Sat Oct 21 05:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.41699999999999998,"15min":0.432},"min":{"1min":0.23200000000000001,"5min":0.215,"15min":0.19800000000000001},"max":{"1min":0.68899999999999995,"5min":3.5649999999999999,"15min":6.4119999999999999},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":0.65100000000000002,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4141854720,"data_stored":4714311680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3992043520,"data_stored":4538920979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8134086656,"data_stored":9253421075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:19:17.966 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:17.966 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:19:37.968 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:19:38.548 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:19:38.549 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:19:38.587 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2324,"stamp":"2023-10-21T05:19:37.732700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28067519720,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2012254,"num_read_kb":7868945,"num_write":1994541,"num_write_kb":7977143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77148,"ondisk_log_size":77148,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20801192,"kb_used_data":15951952,"kb_used_omap":1771,"kb_used_meta":4847444,"kb_avail":262314328,"statfs":{"total":289910292480,"available":268609871872,"internally_reserved":0,"allocated":16334798848,"data_stored":18571713550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1814442,"internal_metadata":4963782742},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,5,1,10,11,3,12,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14479360,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3939,"num_read_kb":15732,"num_write":3988,"num_write_kb":15952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002702"},"pg_stats":[{"pgid":"2.7","version":"86'9830","reported_seq":13562,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:32.068487+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:19:32.068487+0000","last_peered":"2023-10-21T05:19:32.068487+0000","last_clean":"2023-10-21T05:19:32.068487+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:19:32.068487+0000","last_undegraded":"2023-10-21T05:19:32.068487+0000","last_fullsized":"2023-10-21T05:19:32.068487+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9830,"log_dups_size":0,"ondisk_log_size":9830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1005,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:40.494463+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:18:40.494463+0000","last_peered":"2023-10-21T05:18:40.494463+0000","last_clean":"2023-10-21T05:18:40.494463+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:18:40.494463+0000","last_undegraded":"2023-10-21T05:18:40.494463+0000","last_fullsized":"2023-10-21T05:18:40.494463+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'16227","reported_seq":22271,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:32.068978+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:19:32.068978+0000","last_peered":"2023-10-21T05:19:32.068978+0000","last_clean":"2023-10-21T05:19:32.068978+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:19:32.068978+0000","last_undegraded":"2023-10-21T05:19:32.068978+0000","last_fullsized":"2023-10-21T05:19:32.068978+0000","mapping_epoch":12,"log_start":"75'6200","ondisk_log_start":"75'6200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'3359","reported_seq":5203,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:29.549262+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:19:29.549262+0000","last_peered":"2023-10-21T05:19:29.549262+0000","last_clean":"2023-10-21T05:19:29.549262+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:19:29.549262+0000","last_undegraded":"2023-10-21T05:19:29.549262+0000","last_fullsized":"2023-10-21T05:19:29.549262+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3359,"log_dups_size":0,"ondisk_log_size":3359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"86'504566","reported_seq":1005174,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:34.676025+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:19:34.676025+0000","last_peered":"2023-10-21T05:19:34.676025+0000","last_clean":"2023-10-21T05:19:34.676025+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:19:34.676025+0000","last_undegraded":"2023-10-21T05:19:34.676025+0000","last_fullsized":"2023-10-21T05:19:34.676025+0000","mapping_epoch":15,"log_start":"85'494517","ondisk_log_start":"85'494517","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7054430208,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":501929,"num_read_kb":1977212,"num_write":500808,"num_write_kb":2003232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'16639","reported_seq":22909,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:33.491128+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:19:33.491128+0000","last_peered":"2023-10-21T05:19:33.491128+0000","last_clean":"2023-10-21T05:19:33.491128+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:19:33.491128+0000","last_undegraded":"2023-10-21T05:19:33.491128+0000","last_fullsized":"2023-10-21T05:19:33.491128+0000","mapping_epoch":12,"log_start":"75'6600","ondisk_log_start":"75'6600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"86'497940","reported_seq":991018,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:34.118457+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:19:34.118457+0000","last_peered":"2023-10-21T05:19:34.118457+0000","last_clean":"2023-10-21T05:19:34.118457+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:19:34.118457+0000","last_undegraded":"2023-10-21T05:19:34.118457+0000","last_fullsized":"2023-10-21T05:19:34.118457+0000","mapping_epoch":15,"log_start":"85'487919","ondisk_log_start":"85'487919","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1021,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 148s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6958292992,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":494411,"num_read_kb":1946796,"num_write":495557,"num_write_kb":1982228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6527,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:32.067988+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:19:32.067988+0000","last_peered":"2023-10-21T05:19:32.067988+0000","last_clean":"2023-10-21T05:19:32.067988+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:19:32.067988+0000","last_undegraded":"2023-10-21T05:19:32.067988+0000","last_fullsized":"2023-10-21T05:19:32.067988+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"86'518253","reported_seq":1030945,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:34.688027+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:19:34.688027+0000","last_peered":"2023-10-21T05:19:34.688027+0000","last_clean":"2023-10-21T05:19:34.688027+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:19:34.688027+0000","last_undegraded":"2023-10-21T05:19:34.688027+0000","last_fullsized":"2023-10-21T05:19:34.688027+0000","mapping_epoch":15,"log_start":"85'508225","ondisk_log_start":"85'508225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7253721088,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":514066,"num_read_kb":2024880,"num_write":514377,"num_write_kb":2057508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":164,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:40.511776+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:18:40.511776+0000","last_peered":"2023-10-21T05:18:40.511776+0000","last_clean":"2023-10-21T05:18:40.511776+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:18:40.511776+0000","last_undegraded":"2023-10-21T05:18:40.511776+0000","last_fullsized":"2023-10-21T05:18:40.511776+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"86'486624","reported_seq":969047,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:34.129285+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:19:34.129285+0000","last_peered":"2023-10-21T05:19:34.129285+0000","last_clean":"2023-10-21T05:19:34.129285+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:19:34.129285+0000","last_undegraded":"2023-10-21T05:19:34.129285+0000","last_fullsized":"2023-10-21T05:19:34.129285+0000","mapping_epoch":15,"log_start":"85'476618","ondisk_log_start":"85'476618","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6800482323,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":483683,"num_read_kb":1904580,"num_write":482993,"num_write_kb":1931966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"86'3338","reported_seq":4717,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:29.561012+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:19:29.561012+0000","last_peered":"2023-10-21T05:19:29.561012+0000","last_clean":"2023-10-21T05:19:29.561012+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:19:29.561012+0000","last_undegraded":"2023-10-21T05:19:29.561012+0000","last_fullsized":"2023-10-21T05:19:29.561012+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3338,"log_dups_size":0,"ondisk_log_size":3338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":261,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:40.498269+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:18:40.498269+0000","last_peered":"2023-10-21T05:18:40.498269+0000","last_clean":"2023-10-21T05:18:40.498269+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:18:40.498269+0000","last_undegraded":"2023-10-21T05:18:40.498269+0000","last_fullsized":"2023-10-21T05:18:40.498269+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28066926611,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":1994089,"num_read_kb":7853468,"num_write":1993735,"num_write_kb":7974934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16332845056,"data_stored":18571513894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40104,"ondisk_log_size":40104,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":36945,"ondisk_log_size":36945,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5744068,"kb_used_data":4061564,"kb_used_omap":373,"kb_used_meta":1682122,"kb_avail":88627772,"statfs":{"total":96636764160,"available":90754838528,"internally_reserved":0,"allocated":4159041536,"data_stored":4730905834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1722493488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,0,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:19:05 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45500000000000002,"15min":0.40699999999999997},"min":{"1min":0.17799999999999999,"5min":0.16800000000000001,"15min":0.14299999999999999},"max":{"1min":2.6030000000000002,"5min":4.0259999999999998,"15min":4.9889999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.54500000000000004,"15min":0.47899999999999998},"min":{"1min":0.246,"5min":0.182,"15min":0.182},"max":{"1min":6.992,"5min":6.992,"15min":6.992},"last":0.20499999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:19:05 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.17899999999999999},"max":{"1min":2.5019999999999998,"5min":5.4390000000000001,"15min":14.318},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.20899999999999999},"max":{"1min":2.5339999999999998,"5min":5.4740000000000002,"15min":14.343},"last":0.315}]}]},{"osd":2,"up_from":8,"seq":34359739267,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9281192,"kb_used_data":7974816,"kb_used_omap":1038,"kb_used_meta":1305329,"kb_avail":85090648,"statfs":{"total":96636764160,"available":87132823552,"internally_reserved":0,"allocated":8166211584,"data_stored":9285062354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1063420,"internal_metadata":1336657412},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:59 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.16900000000000001,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":7.234,"5min":7.234,"15min":7.234},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.33600000000000002,"15min":0.35799999999999998},"min":{"1min":0.217,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":4.9029999999999996},"last":0.42799999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:18:59 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.39900000000000002,"15min":0.497},"min":{"1min":0.249,"5min":0.23200000000000001,"15min":0.21199999999999999},"max":{"1min":0.38200000000000001,"5min":3.8140000000000001,"15min":26.620999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.41299999999999998,"15min":0.54400000000000004},"min":{"1min":0.27500000000000002,"5min":0.23000000000000001,"15min":0.20000000000000001},"max":{"1min":1.2250000000000001,"5min":3.8370000000000002,"15min":26.596},"last":0.39400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739266,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5775932,"kb_used_data":3915572,"kb_used_omap":359,"kb_used_meta":1859992,"kb_avail":88595908,"statfs":{"total":96636764160,"available":90722209792,"internally_reserved":0,"allocated":4009545728,"data_stored":4555745362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1904631842},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,3,5,3,12,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.45100000000000001,"15min":0.41799999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16400000000000001},"max":{"1min":1.022,"5min":5.0490000000000004,"15min":9.218},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.377,"15min":0.46500000000000002},"min":{"1min":0.20799999999999999,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":0.623,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.41699999999999998,"15min":0.432},"min":{"1min":0.23200000000000001,"5min":0.215,"15min":0.19800000000000001},"max":{"1min":0.68899999999999995,"5min":3.5649999999999999,"15min":6.4119999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":0.65100000000000002,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4158205952,"data_stored":4730662912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4008148992,"data_stored":4555026451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8166490112,"data_stored":9285824531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:19:38.589 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:38.590 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:19:40.805 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:19:40.799+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:19:40.806 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:19:40.799+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:19:40.806 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:19:40.799+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:19:58.591 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:19:59.073 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:19:59.073 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:19:59.111 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2335,"stamp":"2023-10-21T05:19:57.736199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28096896232,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2020464,"num_read_kb":7901705,"num_write":2002603,"num_write_kb":8009391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77167,"ondisk_log_size":77167,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21041608,"kb_used_data":16016112,"kb_used_omap":1771,"kb_used_meta":5023700,"kb_avail":262073912,"statfs":{"total":289910292480,"available":268363685888,"internally_reserved":0,"allocated":16400498688,"data_stored":18637399671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1814442,"internal_metadata":5144268886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,7,13,7,17,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14942208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4256,"num_read_kb":16968,"num_write":4071,"num_write_kb":16284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001667"},"pg_stats":[{"pgid":"2.7","version":"87'9894","reported_seq":13639,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:19:52.547767+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:19:52.547767+0000","last_peered":"2023-10-21T05:19:52.547767+0000","last_clean":"2023-10-21T05:19:52.547767+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:19:52.547767+0000","last_undegraded":"2023-10-21T05:19:52.547767+0000","last_fullsized":"2023-10-21T05:19:52.547767+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1006,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:48.967783+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:18:48.967783+0000","last_peered":"2023-10-21T05:18:48.967783+0000","last_clean":"2023-10-21T05:18:48.967783+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:18:48.967783+0000","last_undegraded":"2023-10-21T05:18:48.967783+0000","last_fullsized":"2023-10-21T05:18:48.967783+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"87'16333","reported_seq":22398,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:19:52.548273+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:19:52.548273+0000","last_peered":"2023-10-21T05:19:52.548273+0000","last_clean":"2023-10-21T05:19:52.548273+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:19:52.548273+0000","last_undegraded":"2023-10-21T05:19:52.548273+0000","last_fullsized":"2023-10-21T05:19:52.548273+0000","mapping_epoch":12,"log_start":"75'6300","ondisk_log_start":"75'6300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"87'3381","reported_seq":5230,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:19:51.587148+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:19:51.587148+0000","last_peered":"2023-10-21T05:19:51.587148+0000","last_clean":"2023-10-21T05:19:51.587148+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:19:51.587148+0000","last_undegraded":"2023-10-21T05:19:51.587148+0000","last_fullsized":"2023-10-21T05:19:51.587148+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"87'506626","reported_seq":1009350,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:19:54.846769+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:19:54.846769+0000","last_peered":"2023-10-21T05:19:54.846769+0000","last_clean":"2023-10-21T05:19:54.846769+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:19:54.846769+0000","last_undegraded":"2023-10-21T05:19:54.846769+0000","last_fullsized":"2023-10-21T05:19:54.846769+0000","mapping_epoch":15,"log_start":"85'496617","ondisk_log_start":"85'496617","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7061893120,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":504043,"num_read_kb":1985640,"num_write":502868,"num_write_kb":2011472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"87'16714","reported_seq":23005,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:19:52.547909+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:19:52.547909+0000","last_peered":"2023-10-21T05:19:52.547909+0000","last_clean":"2023-10-21T05:19:52.547909+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:19:52.547909+0000","last_undegraded":"2023-10-21T05:19:52.547909+0000","last_fullsized":"2023-10-21T05:19:52.547909+0000","mapping_epoch":12,"log_start":"75'6700","ondisk_log_start":"75'6700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"87'500024","reported_seq":995152,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:54.204245+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:19:54.204245+0000","last_peered":"2023-10-21T05:19:54.204245+0000","last_clean":"2023-10-21T05:19:54.204245+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:19:54.204245+0000","last_undegraded":"2023-10-21T05:19:54.204245+0000","last_fullsized":"2023-10-21T05:19:54.204245+0000","mapping_epoch":15,"log_start":"85'490019","ondisk_log_start":"85'490019","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1126,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 168s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6965612544,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":496446,"num_read_kb":1954928,"num_write":497573,"num_write_kb":1990292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6548,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:19:52.558884+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:19:52.558884+0000","last_peered":"2023-10-21T05:19:52.558884+0000","last_clean":"2023-10-21T05:19:52.558884+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:19:52.558884+0000","last_undegraded":"2023-10-21T05:19:52.558884+0000","last_fullsized":"2023-10-21T05:19:52.558884+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"87'520326","reported_seq":1035131,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:19:54.854900+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:19:54.854900+0000","last_peered":"2023-10-21T05:19:54.854900+0000","last_clean":"2023-10-21T05:19:54.854900+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:19:54.854900+0000","last_undegraded":"2023-10-21T05:19:54.854900+0000","last_fullsized":"2023-10-21T05:19:54.854900+0000","mapping_epoch":15,"log_start":"85'510325","ondisk_log_start":"85'510325","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7261356032,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":516177,"num_read_kb":2033296,"num_write":516450,"num_write_kb":2065800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":165,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:29.545107+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:19:29.545107+0000","last_peered":"2023-10-21T05:19:29.545107+0000","last_clean":"2023-10-21T05:19:29.545107+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:19:29.545107+0000","last_undegraded":"2023-10-21T05:19:29.545107+0000","last_fullsized":"2023-10-21T05:19:29.545107+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"87'488537","reported_seq":972909,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:19:54.205510+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:19:54.205510+0000","last_peered":"2023-10-21T05:19:54.205510+0000","last_clean":"2023-10-21T05:19:54.205510+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:19:54.205510+0000","last_undegraded":"2023-10-21T05:19:54.205510+0000","last_fullsized":"2023-10-21T05:19:54.205510+0000","mapping_epoch":15,"log_start":"85'478518","ondisk_log_start":"85'478518","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6807441427,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":485633,"num_read_kb":1912364,"num_write":484906,"num_write_kb":1939618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"87'3360","reported_seq":4744,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:19:51.587758+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:19:51.587758+0000","last_peered":"2023-10-21T05:19:51.587758+0000","last_clean":"2023-10-21T05:19:51.587758+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:19:51.587758+0000","last_undegraded":"2023-10-21T05:19:51.587758+0000","last_fullsized":"2023-10-21T05:19:51.587758+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3360,"log_dups_size":0,"ondisk_log_size":3360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":262,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:24.065454+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:19:24.065454+0000","last_peered":"2023-10-21T05:19:24.065454+0000","last_clean":"2023-10-21T05:19:24.065454+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:19:24.065454+0000","last_undegraded":"2023-10-21T05:19:24.065454+0000","last_fullsized":"2023-10-21T05:19:24.065454+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28096303123,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2002299,"num_read_kb":7886228,"num_write":2001797,"num_write_kb":8007182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16398778368,"data_stored":18637447206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40034,"ondisk_log_size":40034,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37034,"ondisk_log_size":37034,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739272,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5796672,"kb_used_data":4078072,"kb_used_omap":373,"kb_used_meta":1718218,"kb_avail":88575168,"statfs":{"total":96636764160,"available":90700972032,"internally_reserved":0,"allocated":4175945728,"data_stored":4747805453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1759455792},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,4,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:19:05 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45500000000000002,"15min":0.40699999999999997},"min":{"1min":0.17799999999999999,"5min":0.16800000000000001,"15min":0.14299999999999999},"max":{"1min":2.6030000000000002,"5min":4.0259999999999998,"15min":4.9889999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.54500000000000004,"15min":0.47899999999999998},"min":{"1min":0.246,"5min":0.182,"15min":0.182},"max":{"1min":6.992,"5min":6.992,"15min":6.992},"last":0.248}]},{"osd":2,"last update":"Sat Oct 21 05:19:05 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.17899999999999999},"max":{"1min":2.5019999999999998,"5min":5.4390000000000001,"15min":14.318},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.20899999999999999},"max":{"1min":2.5339999999999998,"5min":5.4740000000000002,"15min":14.343},"last":0.33600000000000002}]}]},{"osd":2,"up_from":8,"seq":34359739271,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9399156,"kb_used_data":8006764,"kb_used_omap":1038,"kb_used_meta":1391345,"kb_avail":84972684,"statfs":{"total":96636764160,"available":87012028416,"internally_reserved":0,"allocated":8198926336,"data_stored":9317772533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1063420,"internal_metadata":1424737796},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:59 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.16900000000000001,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":7.234,"5min":7.234,"15min":7.234},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.33600000000000002,"15min":0.35799999999999998},"min":{"1min":0.217,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":4.9029999999999996},"last":0.247}]},{"osd":1,"last update":"Sat Oct 21 05:18:59 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.39900000000000002,"15min":0.497},"min":{"1min":0.249,"5min":0.23200000000000001,"15min":0.21199999999999999},"max":{"1min":0.38200000000000001,"5min":3.8140000000000001,"15min":26.620999999999999},"last":0.221},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.41299999999999998,"15min":0.54400000000000004},"min":{"1min":0.27500000000000002,"5min":0.23000000000000001,"15min":0.20000000000000001},"max":{"1min":1.2250000000000001,"5min":3.8370000000000002,"15min":26.596},"last":0.28000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739270,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5845780,"kb_used_data":3931276,"kb_used_omap":359,"kb_used_meta":1914136,"kb_avail":88526060,"statfs":{"total":96636764160,"available":90650685440,"internally_reserved":0,"allocated":4025626624,"data_stored":4571821685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1960075298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,2,10,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.442,"15min":0.39100000000000001},"min":{"1min":0.20899999999999999,"5min":0.17899999999999999,"15min":0.16400000000000001},"max":{"1min":0.41899999999999998,"5min":5.0490000000000004,"15min":5.0490000000000004},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.375,"15min":0.45800000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.17000000000000001},"max":{"1min":0.40100000000000002,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.24399999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42699999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.63600000000000001,"5min":3.5649999999999999,"15min":6.4119999999999999},"last":0.308},{"interface":"front","average":{"1min":1.097,"5min":0.54200000000000004,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":5.6050000000000004,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":5.8179999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4175130624,"data_stored":4747587584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4024238080,"data_stored":4571115539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8199409664,"data_stored":9318744083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:19:59.112 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:59.113 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:19:59.625 INFO:teuthology.orchestra.run.smithi119.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:19:59.667 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:20:04.902 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:20:04.895+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b130a90 2023-10-21T05:20:04.903 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:20:04.895+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T05:20:04.903 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:20:04.895+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dcc30 2023-10-21T05:20:04.903 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:20:04.895+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:20:10.326 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:10.319+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:20:15.282 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.278+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:20:15.293 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.290+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:20:15.310 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.306+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6f790 2023-10-21T05:20:15.310 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.306+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:20:15.310 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.306+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T05:20:15.311 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.306+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:20:15.311 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.306+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:20:15.311 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.306+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:20:15.312 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.306+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:20:15.616 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.614+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:20:15.626 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.622+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:20:15.627 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.622+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:20:15.627 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:15.622+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:20:18.028 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:18.022+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:20:18.031 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:18.030+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:20:18.031 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:18.030+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:20:18.031 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:18.030+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:20:18.034 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:18.030+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:20:18.036 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:18.034+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:20:18.045 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:18.042+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:20:18.046 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:18.042+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:20:18.046 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:18.042+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:20:19.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:20:20.086 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:20:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:20:20.122 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2346,"stamp":"2023-10-21T05:20:19.748223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28129905896,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2029934,"num_read_kb":7939477,"num_write":2011677,"num_write_kb":8045687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77605,"ondisk_log_size":77605,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21294292,"kb_used_data":16092412,"kb_used_omap":1771,"kb_used_meta":5200084,"kb_avail":261821228,"statfs":{"total":289910292480,"available":268104937472,"internally_reserved":0,"allocated":16478629888,"data_stored":18715530871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1814442,"internal_metadata":5324886102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,4,4,6,8,11,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18386944,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5133,"num_read_kb":20476,"num_write":5074,"num_write_kb":20296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010370"},"pg_stats":[{"pgid":"2.7","version":"87'9948","reported_seq":13708,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:18.389305+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:20:18.389305+0000","last_peered":"2023-10-21T05:20:18.389305+0000","last_clean":"2023-10-21T05:20:18.389305+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:20:18.389305+0000","last_undegraded":"2023-10-21T05:20:18.389305+0000","last_fullsized":"2023-10-21T05:20:18.389305+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9948,"log_dups_size":0,"ondisk_log_size":9948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1006,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:48.967783+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:18:48.967783+0000","last_peered":"2023-10-21T05:18:48.967783+0000","last_clean":"2023-10-21T05:18:48.967783+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:18:48.967783+0000","last_undegraded":"2023-10-21T05:18:48.967783+0000","last_fullsized":"2023-10-21T05:18:48.967783+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"87'16417","reported_seq":22507,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:18.408898+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:20:18.408898+0000","last_peered":"2023-10-21T05:20:18.408898+0000","last_clean":"2023-10-21T05:20:18.408898+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:20:18.408898+0000","last_undegraded":"2023-10-21T05:20:18.408898+0000","last_fullsized":"2023-10-21T05:20:18.408898+0000","mapping_epoch":12,"log_start":"75'6400","ondisk_log_start":"75'6400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"87'3400","reported_seq":5254,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:18.147589+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:20:18.147589+0000","last_peered":"2023-10-21T05:20:18.147589+0000","last_clean":"2023-10-21T05:20:18.147589+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:20:18.147589+0000","last_undegraded":"2023-10-21T05:20:18.147589+0000","last_fullsized":"2023-10-21T05:20:18.147589+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3400,"log_dups_size":0,"ondisk_log_size":3400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"87'508706","reported_seq":1013566,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:14.921138+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:20:14.921138+0000","last_peered":"2023-10-21T05:20:14.921138+0000","last_clean":"2023-10-21T05:20:14.921138+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:20:14.921138+0000","last_undegraded":"2023-10-21T05:20:14.921138+0000","last_fullsized":"2023-10-21T05:20:14.921138+0000","mapping_epoch":15,"log_start":"85'498617","ondisk_log_start":"85'498617","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7069392896,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":506182,"num_read_kb":1994172,"num_write":504948,"num_write_kb":2019792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"87'16799","reported_seq":23110,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:14.553733+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:20:14.553733+0000","last_peered":"2023-10-21T05:20:14.553733+0000","last_clean":"2023-10-21T05:20:14.553733+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:20:14.553733+0000","last_undegraded":"2023-10-21T05:20:14.553733+0000","last_fullsized":"2023-10-21T05:20:14.553733+0000","mapping_epoch":12,"log_start":"75'6700","ondisk_log_start":"75'6700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"87'502612","reported_seq":1000366,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:19.312824+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:20:19.312824+0000","last_peered":"2023-10-21T05:20:19.312824+0000","last_clean":"2023-10-21T05:20:19.312824+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:20:19.312824+0000","last_undegraded":"2023-10-21T05:20:19.312824+0000","last_fullsized":"2023-10-21T05:20:19.312824+0000","mapping_epoch":15,"log_start":"86'492519","ondisk_log_start":"86'492519","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1289,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 193s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6974640128,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":499050,"num_read_kb":1965332,"num_write":500059,"num_write_kb":2000236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6574,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:18.148996+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:20:18.148996+0000","last_peered":"2023-10-21T05:20:18.148996+0000","last_clean":"2023-10-21T05:20:18.148996+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:20:18.148996+0000","last_undegraded":"2023-10-21T05:20:18.148996+0000","last_fullsized":"2023-10-21T05:20:18.148996+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"87'522379","reported_seq":1039400,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:14.912755+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:20:14.912755+0000","last_peered":"2023-10-21T05:20:14.912755+0000","last_clean":"2023-10-21T05:20:14.912755+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:20:14.912755+0000","last_undegraded":"2023-10-21T05:20:14.912755+0000","last_fullsized":"2023-10-21T05:20:14.912755+0000","mapping_epoch":15,"log_start":"85'512325","ondisk_log_start":"85'512325","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7268823040,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":518390,"num_read_kb":2042116,"num_write":518503,"num_write_kb":2074012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":165,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:29.545107+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:19:29.545107+0000","last_peered":"2023-10-21T05:19:29.545107+0000","last_clean":"2023-10-21T05:19:29.545107+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:19:29.545107+0000","last_undegraded":"2023-10-21T05:19:29.545107+0000","last_fullsized":"2023-10-21T05:19:29.545107+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"87'490992","reported_seq":977880,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:19.303552+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:20:19.303552+0000","last_peered":"2023-10-21T05:20:19.303552+0000","last_clean":"2023-10-21T05:20:19.303552+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:20:19.303552+0000","last_undegraded":"2023-10-21T05:20:19.303552+0000","last_fullsized":"2023-10-21T05:20:19.303552+0000","mapping_epoch":15,"log_start":"85'480918","ondisk_log_start":"85'480918","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6816456723,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":488147,"num_read_kb":1922380,"num_write":487361,"num_write_kb":1949438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"87'3380","reported_seq":4769,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:18.147460+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:20:18.147460+0000","last_peered":"2023-10-21T05:20:18.147460+0000","last_clean":"2023-10-21T05:20:18.147460+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:20:18.147460+0000","last_undegraded":"2023-10-21T05:20:18.147460+0000","last_fullsized":"2023-10-21T05:20:18.147460+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3380,"log_dups_size":0,"ondisk_log_size":3380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":262,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:24.065454+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:19:24.065454+0000","last_peered":"2023-10-21T05:19:24.065454+0000","last_clean":"2023-10-21T05:19:24.065454+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:19:24.065454+0000","last_undegraded":"2023-10-21T05:19:24.065454+0000","last_fullsized":"2023-10-21T05:19:24.065454+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28129312787,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2011769,"num_read_kb":7924000,"num_write":2010871,"num_write_kb":8043478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16477343744,"data_stored":18716012582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40310,"ondisk_log_size":40310,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37196,"ondisk_log_size":37196,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739276,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5849300,"kb_used_data":4094604,"kb_used_omap":373,"kb_used_meta":1754314,"kb_avail":88522540,"statfs":{"total":96636764160,"available":90647080960,"internally_reserved":0,"allocated":4192874496,"data_stored":4764734221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1796418096},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.41599999999999998},"min":{"1min":0.17999999999999999,"5min":0.16800000000000001,"15min":0.14299999999999999},"max":{"1min":3.3399999999999999,"5min":4.0259999999999998,"15min":4.9889999999999999},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.54900000000000004,"15min":0.47199999999999998},"min":{"1min":0.20499999999999999,"5min":0.182,"15min":0.182},"max":{"1min":3.2770000000000001,"5min":6.992,"15min":6.992},"last":0.67800000000000005}]},{"osd":2,"last update":"Sat Oct 21 05:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.57299999999999995,"15min":0.54400000000000004},"min":{"1min":0.21299999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.0539999999999998,"5min":5.4390000000000001,"15min":14.318},"last":3.4420000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.61699999999999999,"15min":0.58899999999999997},"min":{"1min":0.28499999999999998,"5min":0.26200000000000001,"15min":0.245},"max":{"1min":3.0859999999999999,"5min":5.4740000000000002,"15min":14.343},"last":3.464}]}]},{"osd":2,"up_from":8,"seq":34359739276,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9539284,"kb_used_data":8046860,"kb_used_omap":1038,"kb_used_meta":1491377,"kb_avail":84832556,"statfs":{"total":96636764160,"available":86868537344,"internally_reserved":0,"allocated":8239984640,"data_stored":9358830837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1063420,"internal_metadata":1527170564},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.436,"15min":0.47599999999999998},"min":{"1min":0.182,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":1.127,"5min":7.234,"15min":7.234},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.34999999999999998,"15min":0.35099999999999998},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":2.6699999999999999,"5min":2.6699999999999999,"15min":4.9029999999999996},"last":0.22}]},{"osd":1,"last update":"Sat Oct 21 05:20:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.38500000000000001,"15min":0.498},"min":{"1min":0.219,"5min":0.219,"15min":0.21199999999999999},"max":{"1min":0.67700000000000005,"5min":3.8140000000000001,"15min":26.620999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.433,"5min":0.41399999999999998,"15min":0.51500000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.20000000000000001},"max":{"1min":2.722,"5min":3.8370000000000002,"15min":26.596},"last":0.314}]}]},{"osd":1,"up_from":8,"seq":34359739275,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5905708,"kb_used_data":3950948,"kb_used_omap":359,"kb_used_meta":1954392,"kb_avail":88466132,"statfs":{"total":96636764160,"available":90589319168,"internally_reserved":0,"allocated":4045770752,"data_stored":4591965813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":2001297442},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,4,5,11,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.442,"15min":0.39100000000000001},"min":{"1min":0.20899999999999999,"5min":0.17899999999999999,"15min":0.16400000000000001},"max":{"1min":0.41899999999999998,"5min":5.0490000000000004,"15min":5.0490000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.375,"15min":0.45800000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.17000000000000001},"max":{"1min":0.40100000000000002,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.33700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42699999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.63600000000000001,"5min":3.5649999999999999,"15min":6.4119999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.097,"5min":0.54200000000000004,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":5.6050000000000004,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.314}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4192067584,"data_stored":4764524544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4044472320,"data_stored":4591349779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8240803840,"data_stored":9360138259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:20:20.124 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:20.125 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:20:24.461 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:20:24.458+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:20:34.277 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:20:34.266+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:20:40.126 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:20:40.588 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:20:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:20:40.661 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2356,"stamp":"2023-10-21T05:20:39.752565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28158639336,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2038151,"num_read_kb":7972237,"num_write":2019514,"num_write_kb":8077035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77668,"ondisk_log_size":77668,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21509280,"kb_used_data":16148232,"kb_used_omap":1771,"kb_used_meta":5359252,"kb_avail":261606240,"statfs":{"total":289910292480,"available":267884789760,"internally_reserved":0,"allocated":16535789568,"data_stored":18772690551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1814442,"internal_metadata":5487874134},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,9,39,12,26,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18116608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5220,"num_read_kb":20808,"num_write":4945,"num_write_kb":19780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003002"},"pg_stats":[{"pgid":"2.7","version":"87'9991","reported_seq":13761,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:39.021057+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:20:39.021057+0000","last_peered":"2023-10-21T05:20:39.021057+0000","last_clean":"2023-10-21T05:20:39.021057+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:20:39.021057+0000","last_undegraded":"2023-10-21T05:20:39.021057+0000","last_fullsized":"2023-10-21T05:20:39.021057+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":9991,"log_dups_size":0,"ondisk_log_size":9991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1006,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:18:48.967783+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:18:48.967783+0000","last_peered":"2023-10-21T05:18:48.967783+0000","last_clean":"2023-10-21T05:18:48.967783+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:18:48.967783+0000","last_undegraded":"2023-10-21T05:18:48.967783+0000","last_fullsized":"2023-10-21T05:18:48.967783+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"87'16486","reported_seq":22593,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:39.039990+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:20:39.039990+0000","last_peered":"2023-10-21T05:20:39.039990+0000","last_clean":"2023-10-21T05:20:39.039990+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:20:39.039990+0000","last_undegraded":"2023-10-21T05:20:39.039990+0000","last_fullsized":"2023-10-21T05:20:39.039990+0000","mapping_epoch":12,"log_start":"75'6400","ondisk_log_start":"75'6400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"87'3414","reported_seq":5272,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:38.967778+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:20:38.967778+0000","last_peered":"2023-10-21T05:20:38.967778+0000","last_clean":"2023-10-21T05:20:38.967778+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:20:38.967778+0000","last_undegraded":"2023-10-21T05:20:38.967778+0000","last_fullsized":"2023-10-21T05:20:38.967778+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3414,"log_dups_size":0,"ondisk_log_size":3414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"87'510601","reported_seq":1017536,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:34.981264+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:20:34.981264+0000","last_peered":"2023-10-21T05:20:34.981264+0000","last_clean":"2023-10-21T05:20:34.981264+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:20:34.981264+0000","last_undegraded":"2023-10-21T05:20:34.981264+0000","last_fullsized":"2023-10-21T05:20:34.981264+0000","mapping_epoch":15,"log_start":"86'500517","ondisk_log_start":"86'500517","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7076397056,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":508257,"num_read_kb":2002440,"num_write":506843,"num_write_kb":2027372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"87'16875","reported_seq":23207,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:34.428356+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:20:34.428356+0000","last_peered":"2023-10-21T05:20:34.428356+0000","last_clean":"2023-10-21T05:20:34.428356+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:20:34.428356+0000","last_undegraded":"2023-10-21T05:20:34.428356+0000","last_fullsized":"2023-10-21T05:20:34.428356+0000","mapping_epoch":12,"log_start":"75'6800","ondisk_log_start":"75'6800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"87'504741","reported_seq":1004532,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:39.358932+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:20:39.358932+0000","last_peered":"2023-10-21T05:20:39.358932+0000","last_clean":"2023-10-21T05:20:39.358932+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:20:39.358932+0000","last_undegraded":"2023-10-21T05:20:39.358932+0000","last_fullsized":"2023-10-21T05:20:39.358932+0000","mapping_epoch":15,"log_start":"86'494719","ondisk_log_start":"86'494719","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1456,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 213s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6982025216,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":501081,"num_read_kb":1973432,"num_write":502079,"num_write_kb":2008316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6589,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:33.508610+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:20:33.508610+0000","last_peered":"2023-10-21T05:20:33.508610+0000","last_clean":"2023-10-21T05:20:33.508610+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:20:33.508610+0000","last_undegraded":"2023-10-21T05:20:33.508610+0000","last_fullsized":"2023-10-21T05:20:33.508610+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"87'524396","reported_seq":1043517,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:34.993622+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:20:34.993622+0000","last_peered":"2023-10-21T05:20:34.993622+0000","last_clean":"2023-10-21T05:20:34.993622+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:20:34.993622+0000","last_undegraded":"2023-10-21T05:20:34.993622+0000","last_fullsized":"2023-10-21T05:20:34.993622+0000","mapping_epoch":15,"log_start":"86'514325","ondisk_log_start":"86'514325","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7276199936,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":520494,"num_read_kb":2050504,"num_write":520520,"num_write_kb":2082080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":165,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:29.545107+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:19:29.545107+0000","last_peered":"2023-10-21T05:19:29.545107+0000","last_clean":"2023-10-21T05:19:29.545107+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:19:29.545107+0000","last_undegraded":"2023-10-21T05:19:29.545107+0000","last_fullsized":"2023-10-21T05:19:29.545107+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"87'492897","reported_seq":981790,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:39.364187+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:20:39.364187+0000","last_peered":"2023-10-21T05:20:39.364187+0000","last_clean":"2023-10-21T05:20:39.364187+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:20:39.364187+0000","last_undegraded":"2023-10-21T05:20:39.364187+0000","last_fullsized":"2023-10-21T05:20:39.364187+0000","mapping_epoch":15,"log_start":"86'482818","ondisk_log_start":"86'482818","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6823424019,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":490154,"num_read_kb":1930384,"num_write":489266,"num_write_kb":1957058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"87'3395","reported_seq":4789,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:38.979163+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:20:38.979163+0000","last_peered":"2023-10-21T05:20:38.979163+0000","last_clean":"2023-10-21T05:20:38.979163+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:20:38.979163+0000","last_undegraded":"2023-10-21T05:20:38.979163+0000","last_fullsized":"2023-10-21T05:20:38.979163+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3395,"log_dups_size":0,"ondisk_log_size":3395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":262,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:24.065454+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:19:24.065454+0000","last_peered":"2023-10-21T05:19:24.065454+0000","last_clean":"2023-10-21T05:19:24.065454+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:19:24.065454+0000","last_undegraded":"2023-10-21T05:19:24.065454+0000","last_fullsized":"2023-10-21T05:19:24.065454+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28158046227,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2019986,"num_read_kb":7956760,"num_write":2018708,"num_write_kb":8074826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16533110784,"data_stored":18771779622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40256,"ondisk_log_size":40256,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37313,"ondisk_log_size":37313,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5907560,"kb_used_data":4110304,"kb_used_omap":373,"kb_used_meta":1796874,"kb_avail":88464280,"statfs":{"total":96636764160,"available":90587422720,"internally_reserved":0,"allocated":4208951296,"data_stored":4780811021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1839999536},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.41599999999999998},"min":{"1min":0.17999999999999999,"5min":0.16800000000000001,"15min":0.14299999999999999},"max":{"1min":3.3399999999999999,"5min":4.0259999999999998,"15min":4.9889999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.54900000000000004,"15min":0.47199999999999998},"min":{"1min":0.20499999999999999,"5min":0.182,"15min":0.182},"max":{"1min":3.2770000000000001,"5min":6.992,"15min":6.992},"last":0.29999999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.57299999999999995,"15min":0.54400000000000004},"min":{"1min":0.21299999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.0539999999999998,"5min":5.4390000000000001,"15min":14.318},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.61699999999999999,"15min":0.58899999999999997},"min":{"1min":0.28499999999999998,"5min":0.26200000000000001,"15min":0.245},"max":{"1min":3.0859999999999999,"5min":5.4740000000000002,"15min":14.343},"last":0.35999999999999999}]}]},{"osd":2,"up_from":8,"seq":34359739279,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9626280,"kb_used_data":8071392,"kb_used_omap":1038,"kb_used_meta":1553841,"kb_avail":84745560,"statfs":{"total":96636764160,"available":86779453440,"internally_reserved":0,"allocated":8265105408,"data_stored":9383951605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1063420,"internal_metadata":1591133700},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.436,"15min":0.47599999999999998},"min":{"1min":0.182,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":1.127,"5min":7.234,"15min":7.234},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.34999999999999998,"15min":0.35099999999999998},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":2.6699999999999999,"5min":2.6699999999999999,"15min":4.9029999999999996},"last":0.26400000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:20:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.38500000000000001,"15min":0.498},"min":{"1min":0.219,"5min":0.219,"15min":0.21199999999999999},"max":{"1min":0.67700000000000005,"5min":3.8140000000000001,"15min":26.620999999999999},"last":0.308},{"interface":"front","average":{"1min":0.433,"5min":0.41399999999999998,"15min":0.51500000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.20000000000000001},"max":{"1min":2.722,"5min":3.8370000000000002,"15min":26.596},"last":0.29399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739279,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5975440,"kb_used_data":3966536,"kb_used_omap":359,"kb_used_meta":2008536,"kb_avail":88396400,"statfs":{"total":96636764160,"available":90517913600,"internally_reserved":0,"allocated":4061732864,"data_stored":4607927925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":2056740898},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,7,36,5,23,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.42899999999999999,"15min":0.39500000000000002},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":2.0579999999999998,"5min":5.0490000000000004,"15min":5.0490000000000004},"last":0.311},{"interface":"front","average":{"1min":0.38,"5min":0.38300000000000001,"15min":0.44400000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.17000000000000001},"max":{"1min":2.0910000000000002,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.214}]},{"osd":2,"last update":"Sat Oct 21 05:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39200000000000002,"15min":0.42799999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.52600000000000002,"5min":3.5649999999999999,"15min":6.4119999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56899999999999995,"15min":0.44400000000000001},"min":{"1min":0.24299999999999999,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":5.8179999999999996,"5min":5.8179999999999996,"15min":5.8179999999999996},"last":0.35599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4208095232,"data_stored":4780552192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4060545024,"data_stored":4607422483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8264470528,"data_stored":9383804947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:20:40.662 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:40.663 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:00.664 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:21:01.108 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:21:01.109 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:21:01.132 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2367,"stamp":"2023-10-21T05:21:00.535093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28191907048,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2047608,"num_read_kb":8009861,"num_write":2028635,"num_write_kb":8113519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77634,"ondisk_log_size":77634,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21771264,"kb_used_data":16225768,"kb_used_omap":1771,"kb_used_meta":5543700,"kb_avail":261344256,"statfs":{"total":289910292480,"available":267616518144,"internally_reserved":0,"allocated":16615186432,"data_stored":18852087415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1814442,"internal_metadata":5676748886},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,7,7,8,10,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18722816,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5308,"num_read_kb":21136,"num_write":5117,"num_write_kb":20468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.780942"},"pg_stats":[{"pgid":"2.7","version":"87'10030","reported_seq":13812,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:59.107777+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:20:59.107777+0000","last_peered":"2023-10-21T05:20:59.107777+0000","last_clean":"2023-10-21T05:20:59.107777+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:20:59.107777+0000","last_undegraded":"2023-10-21T05:20:59.107777+0000","last_fullsized":"2023-10-21T05:20:59.107777+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1007,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.510335+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:19:50.510335+0000","last_peered":"2023-10-21T05:19:50.510335+0000","last_clean":"2023-10-21T05:19:50.510335+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:19:50.510335+0000","last_undegraded":"2023-10-21T05:19:50.510335+0000","last_fullsized":"2023-10-21T05:19:50.510335+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"87'16551","reported_seq":22678,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:59.108094+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:20:59.108094+0000","last_peered":"2023-10-21T05:20:59.108094+0000","last_clean":"2023-10-21T05:20:59.108094+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:20:59.108094+0000","last_undegraded":"2023-10-21T05:20:59.108094+0000","last_fullsized":"2023-10-21T05:20:59.108094+0000","mapping_epoch":12,"log_start":"75'6500","ondisk_log_start":"75'6500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"87'3427","reported_seq":5289,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:57.223139+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:20:57.223139+0000","last_peered":"2023-10-21T05:20:57.223139+0000","last_clean":"2023-10-21T05:20:57.223139+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:20:57.223139+0000","last_undegraded":"2023-10-21T05:20:57.223139+0000","last_fullsized":"2023-10-21T05:20:57.223139+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3427,"log_dups_size":0,"ondisk_log_size":3427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"87'513078","reported_seq":1022646,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.103630+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:21:00.103630+0000","last_peered":"2023-10-21T05:21:00.103630+0000","last_clean":"2023-10-21T05:21:00.103630+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:21:00.103630+0000","last_undegraded":"2023-10-21T05:21:00.103630+0000","last_fullsized":"2023-10-21T05:21:00.103630+0000","mapping_epoch":15,"log_start":"86'503017","ondisk_log_start":"86'503017","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7085486080,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":510887,"num_read_kb":2012884,"num_write":509320,"num_write_kb":2037280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"87'17006","reported_seq":23363,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:59.108176+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:20:59.108176+0000","last_peered":"2023-10-21T05:20:59.108176+0000","last_clean":"2023-10-21T05:20:59.108176+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:20:59.108176+0000","last_undegraded":"2023-10-21T05:20:59.108176+0000","last_fullsized":"2023-10-21T05:20:59.108176+0000","mapping_epoch":12,"log_start":"75'7000","ondisk_log_start":"75'7000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"87'506772","reported_seq":1008662,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:59.441576+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:20:59.441576+0000","last_peered":"2023-10-21T05:20:59.441576+0000","last_clean":"2023-10-21T05:20:59.441576+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:20:59.441576+0000","last_undegraded":"2023-10-21T05:20:59.441576+0000","last_fullsized":"2023-10-21T05:20:59.441576+0000","mapping_epoch":15,"log_start":"86'496719","ondisk_log_start":"86'496719","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1591,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 233s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6989021184,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":503156,"num_read_kb":1981680,"num_write":504026,"num_write_kb":2016104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6614,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:59.118995+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:20:59.118995+0000","last_peered":"2023-10-21T05:20:59.118995+0000","last_clean":"2023-10-21T05:20:59.118995+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:20:59.118995+0000","last_undegraded":"2023-10-21T05:20:59.118995+0000","last_fullsized":"2023-10-21T05:20:59.118995+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"87'527107","reported_seq":1048959,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.070496+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:21:00.070496+0000","last_peered":"2023-10-21T05:21:00.070496+0000","last_clean":"2023-10-21T05:21:00.070496+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:21:00.070496+0000","last_undegraded":"2023-10-21T05:21:00.070496+0000","last_fullsized":"2023-10-21T05:21:00.070496+0000","mapping_epoch":15,"log_start":"86'517025","ondisk_log_start":"86'517025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7286140928,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":523222,"num_read_kb":2061372,"num_write":523231,"num_write_kb":2092924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":165,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:29.545107+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:19:29.545107+0000","last_peered":"2023-10-21T05:19:29.545107+0000","last_clean":"2023-10-21T05:19:29.545107+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:19:29.545107+0000","last_undegraded":"2023-10-21T05:19:29.545107+0000","last_fullsized":"2023-10-21T05:19:29.545107+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"87'494883","reported_seq":985801,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:59.478443+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:20:59.478443+0000","last_peered":"2023-10-21T05:20:59.478443+0000","last_clean":"2023-10-21T05:20:59.478443+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:20:59.478443+0000","last_undegraded":"2023-10-21T05:20:59.478443+0000","last_fullsized":"2023-10-21T05:20:59.478443+0000","mapping_epoch":15,"log_start":"86'484818","ondisk_log_start":"86'484818","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6830665747,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":492178,"num_read_kb":1938448,"num_write":491252,"num_write_kb":1965002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"87'3408","reported_seq":4806,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:57.261658+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:20:57.261658+0000","last_peered":"2023-10-21T05:20:57.261658+0000","last_clean":"2023-10-21T05:20:57.261658+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:20:57.261658+0000","last_undegraded":"2023-10-21T05:20:57.261658+0000","last_fullsized":"2023-10-21T05:20:57.261658+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3408,"log_dups_size":0,"ondisk_log_size":3408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":262,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:19:24.065454+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:19:24.065454+0000","last_peered":"2023-10-21T05:19:24.065454+0000","last_clean":"2023-10-21T05:19:24.065454+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:19:24.065454+0000","last_undegraded":"2023-10-21T05:19:24.065454+0000","last_fullsized":"2023-10-21T05:19:24.065454+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28191313939,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2029443,"num_read_kb":7994384,"num_write":2027829,"num_write_kb":8111310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16612081664,"data_stored":18850750502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40261,"ondisk_log_size":40261,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37274,"ondisk_log_size":37274,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739285,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5991352,"kb_used_data":4131632,"kb_used_omap":373,"kb_used_meta":1859338,"kb_avail":88380488,"statfs":{"total":96636764160,"available":90501619712,"internally_reserved":0,"allocated":4230791168,"data_stored":4802650893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1903962672},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.41599999999999998},"min":{"1min":0.17999999999999999,"5min":0.16800000000000001,"15min":0.14299999999999999},"max":{"1min":3.3399999999999999,"5min":4.0259999999999998,"15min":4.9889999999999999},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.54900000000000004,"15min":0.47199999999999998},"min":{"1min":0.20499999999999999,"5min":0.182,"15min":0.182},"max":{"1min":3.2770000000000001,"5min":6.992,"15min":6.992},"last":0.214}]},{"osd":2,"last update":"Sat Oct 21 05:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.57299999999999995,"15min":0.54400000000000004},"min":{"1min":0.21299999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.0539999999999998,"5min":5.4390000000000001,"15min":14.318},"last":2.113},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.61699999999999999,"15min":0.58899999999999997},"min":{"1min":0.28499999999999998,"5min":0.26200000000000001,"15min":0.245},"max":{"1min":3.0859999999999999,"5min":5.4740000000000002,"15min":14.343},"last":2.1349999999999998}]}]},{"osd":2,"up_from":8,"seq":34359739284,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9770760,"kb_used_data":8111936,"kb_used_omap":1038,"kb_used_meta":1657777,"kb_avail":84601080,"statfs":{"total":96636764160,"available":86631505920,"internally_reserved":0,"allocated":8306622464,"data_stored":9425468661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1063420,"internal_metadata":1697564164},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.436,"15min":0.47599999999999998},"min":{"1min":0.182,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":1.127,"5min":7.234,"15min":7.234},"last":1.03},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.34999999999999998,"15min":0.35099999999999998},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":2.6699999999999999,"5min":2.6699999999999999,"15min":4.9029999999999996},"last":0.26100000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:20:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.38500000000000001,"15min":0.498},"min":{"1min":0.219,"5min":0.219,"15min":0.21199999999999999},"max":{"1min":0.67700000000000005,"5min":3.8140000000000001,"15min":26.620999999999999},"last":0.316},{"interface":"front","average":{"1min":0.433,"5min":0.41399999999999998,"15min":0.51500000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.20000000000000001},"max":{"1min":2.722,"5min":3.8370000000000002,"15min":26.596},"last":0.30099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739283,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6009152,"kb_used_data":3982200,"kb_used_omap":359,"kb_used_meta":2026584,"kb_avail":88362688,"statfs":{"total":96636764160,"available":90483392512,"internally_reserved":0,"allocated":4077772800,"data_stored":4623967861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":2075222050},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,7,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.42899999999999999,"15min":0.39500000000000002},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":2.0579999999999998,"5min":5.0490000000000004,"15min":5.0490000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.38300000000000001,"15min":0.44400000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.17000000000000001},"max":{"1min":2.0910000000000002,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39200000000000002,"15min":0.42799999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.52600000000000002,"5min":3.5649999999999999,"15min":6.4119999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56899999999999995,"15min":0.44400000000000001},"min":{"1min":0.24299999999999999,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":5.8179999999999996,"5min":5.8179999999999996,"15min":5.8179999999999996},"last":0.27000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4229357568,"data_stored":4801814528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4076662784,"data_stored":4623540243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8306061312,"data_stored":9425395731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:21:01.133 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:01.134 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:21.135 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:21:21.566 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:21:21.568 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:21:21.591 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2377,"stamp":"2023-10-21T05:21:19.758953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28217527528,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2054618,"num_read_kb":8037761,"num_write":2035700,"num_write_kb":8141779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77659,"ondisk_log_size":77659,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21789992,"kb_used_data":16277068,"kb_used_omap":1771,"kb_used_meta":5511124,"kb_avail":261325528,"statfs":{"total":289910292480,"available":267597340672,"internally_reserved":0,"allocated":16667717632,"data_stored":18904604767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1814442,"internal_metadata":5643391062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,4,9,15,11,17,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18112512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4934,"num_read_kb":19620,"num_write":4995,"num_write_kb":19980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001953"},"pg_stats":[{"pgid":"2.7","version":"88'10057","reported_seq":13850,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:18.791275+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:21:18.791275+0000","last_peered":"2023-10-21T05:21:18.791275+0000","last_clean":"2023-10-21T05:21:18.791275+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:21:18.791275+0000","last_undegraded":"2023-10-21T05:21:18.791275+0000","last_fullsized":"2023-10-21T05:21:18.791275+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1008,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:59.478791+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:20:59.478791+0000","last_peered":"2023-10-21T05:20:59.478791+0000","last_clean":"2023-10-21T05:20:59.478791+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:20:59.478791+0000","last_undegraded":"2023-10-21T05:20:59.478791+0000","last_fullsized":"2023-10-21T05:20:59.478791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"88'16596","reported_seq":22739,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:18.815141+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:21:18.815141+0000","last_peered":"2023-10-21T05:21:18.815141+0000","last_clean":"2023-10-21T05:21:18.815141+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:21:18.815141+0000","last_undegraded":"2023-10-21T05:21:18.815141+0000","last_fullsized":"2023-10-21T05:21:18.815141+0000","mapping_epoch":12,"log_start":"75'6500","ondisk_log_start":"75'6500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"88'3436","reported_seq":5302,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:18.171291+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:21:18.171291+0000","last_peered":"2023-10-21T05:21:18.171291+0000","last_clean":"2023-10-21T05:21:18.171291+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:21:18.171291+0000","last_undegraded":"2023-10-21T05:21:18.171291+0000","last_fullsized":"2023-10-21T05:21:18.171291+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3436,"log_dups_size":0,"ondisk_log_size":3436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"88'514682","reported_seq":1025730,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:15.181524+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:21:15.181524+0000","last_peered":"2023-10-21T05:21:15.181524+0000","last_clean":"2023-10-21T05:21:15.181524+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:21:15.181524+0000","last_undegraded":"2023-10-21T05:21:15.181524+0000","last_fullsized":"2023-10-21T05:21:15.181524+0000","mapping_epoch":15,"log_start":"86'504617","ondisk_log_start":"86'504617","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7091294208,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":512366,"num_read_kb":2018756,"num_write":510924,"num_write_kb":2043696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"88'17021","reported_seq":23394,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:14.938934+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:21:14.938934+0000","last_peered":"2023-10-21T05:21:14.938934+0000","last_clean":"2023-10-21T05:21:14.938934+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:21:14.938934+0000","last_undegraded":"2023-10-21T05:21:14.938934+0000","last_fullsized":"2023-10-21T05:21:14.938934+0000","mapping_epoch":12,"log_start":"75'7000","ondisk_log_start":"75'7000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"88'508747","reported_seq":1012607,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:19.531246+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:21:19.531246+0000","last_peered":"2023-10-21T05:21:19.531246+0000","last_clean":"2023-10-21T05:21:19.531246+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:21:19.531246+0000","last_undegraded":"2023-10-21T05:21:19.531246+0000","last_fullsized":"2023-10-21T05:21:19.531246+0000","mapping_epoch":15,"log_start":"86'498719","ondisk_log_start":"86'498719","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1685,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 253s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":6996000768,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":505113,"num_read_kb":1989472,"num_write":505946,"num_write_kb":2023784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6630,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:14.491335+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:21:14.491335+0000","last_peered":"2023-10-21T05:21:14.491335+0000","last_clean":"2023-10-21T05:21:14.491335+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:21:14.491335+0000","last_undegraded":"2023-10-21T05:21:14.491335+0000","last_fullsized":"2023-10-21T05:21:14.491335+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"88'528660","reported_seq":1052071,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:15.167466+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:21:15.167466+0000","last_peered":"2023-10-21T05:21:15.167466+0000","last_clean":"2023-10-21T05:21:15.167466+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:21:15.167466+0000","last_undegraded":"2023-10-21T05:21:15.167466+0000","last_fullsized":"2023-10-21T05:21:15.167466+0000","mapping_epoch":15,"log_start":"86'518625","ondisk_log_start":"86'518625","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7291744256,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":524779,"num_read_kb":2067572,"num_write":524784,"num_write_kb":2099136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":166,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.533776+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:19:50.533776+0000","last_peered":"2023-10-21T05:19:50.533776+0000","last_clean":"2023-10-21T05:19:50.533776+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:19:50.533776+0000","last_undegraded":"2023-10-21T05:19:50.533776+0000","last_fullsized":"2023-10-21T05:19:50.533776+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"88'496871","reported_seq":989809,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:19.551048+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:21:19.551048+0000","last_peered":"2023-10-21T05:21:19.551048+0000","last_clean":"2023-10-21T05:21:19.551048+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:21:19.551048+0000","last_undegraded":"2023-10-21T05:21:19.551048+0000","last_fullsized":"2023-10-21T05:21:19.551048+0000","mapping_epoch":15,"log_start":"86'486818","ondisk_log_start":"86'486818","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6837895187,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":494195,"num_read_kb":1946484,"num_write":493240,"num_write_kb":1972954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'3417","reported_seq":4819,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:18.186986+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:21:18.186986+0000","last_peered":"2023-10-21T05:21:18.186986+0000","last_clean":"2023-10-21T05:21:18.186986+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:21:18.186986+0000","last_undegraded":"2023-10-21T05:21:18.186986+0000","last_fullsized":"2023-10-21T05:21:18.186986+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":263,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.521889+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:19:50.521889+0000","last_peered":"2023-10-21T05:19:50.521889+0000","last_clean":"2023-10-21T05:19:50.521889+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:19:50.521889+0000","last_undegraded":"2023-10-21T05:19:50.521889+0000","last_fullsized":"2023-10-21T05:19:50.521889+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28216934419,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2036453,"num_read_kb":8022284,"num_write":2034894,"num_write_kb":8139570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16665694208,"data_stored":18904363046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40181,"ondisk_log_size":40181,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37379,"ondisk_log_size":37379,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739288,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6012928,"kb_used_data":4143480,"kb_used_omap":373,"kb_used_meta":1869066,"kb_avail":88358912,"statfs":{"total":96636764160,"available":90479525888,"internally_reserved":0,"allocated":4242923520,"data_stored":4814778629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1913924144},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:21:14 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.46700000000000003,"15min":0.39700000000000002},"min":{"1min":0.161,"5min":0.161,"15min":0.14299999999999999},"max":{"1min":0.72699999999999998,"5min":4.0259999999999998,"15min":4.9889999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56699999999999995,"15min":0.47499999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.182},"max":{"1min":5.923,"5min":6.992,"15min":6.992},"last":0.25600000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:21:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.61699999999999999,"15min":0.55800000000000005},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":3.4420000000000002,"5min":5.4390000000000001,"15min":14.318},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.65900000000000003,"15min":0.60199999999999998},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.245},"max":{"1min":3.464,"5min":5.4740000000000002,"15min":14.343},"last":0.36499999999999999}]}]},{"osd":2,"up_from":8,"seq":34359739287,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9836280,"kb_used_data":8135792,"kb_used_omap":1038,"kb_used_meta":1699441,"kb_avail":84535560,"statfs":{"total":96636764160,"available":86564413440,"internally_reserved":0,"allocated":8331051008,"data_stored":9449892589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1063420,"internal_metadata":1740228100},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:03 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.438,"15min":0.48199999999999998},"min":{"1min":0.19400000000000001,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":1.03,"5min":7.234,"15min":7.234},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.40000000000000002,"15min":0.34999999999999998},"min":{"1min":0.214,"5min":0.191,"15min":0.191},"max":{"1min":4.4420000000000002,"5min":4.4420000000000002,"15min":4.4420000000000002},"last":0.41999999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:21:03 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.35399999999999998,"15min":0.40100000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.21199999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":4.0510000000000002},"last":0.376},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.38600000000000001,"15min":0.41699999999999998},"min":{"1min":0.28199999999999997,"5min":0.22800000000000001,"15min":0.20000000000000001},"max":{"1min":0.91500000000000004,"5min":2.722,"15min":3.8370000000000002},"last":0.45200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739287,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5940784,"kb_used_data":3997796,"kb_used_omap":359,"kb_used_meta":1942616,"kb_avail":88431056,"statfs":{"total":96636764160,"available":90553401344,"internally_reserved":0,"allocated":4093743104,"data_stored":4639933549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":1989238818},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,4,6,5,13,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.42899999999999999,"15min":0.39500000000000002},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":2.0579999999999998,"5min":5.0490000000000004,"15min":5.0490000000000004},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.38300000000000001,"15min":0.44400000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.17000000000000001},"max":{"1min":2.0910000000000002,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.222}]},{"osd":2,"last update":"Sat Oct 21 05:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39200000000000002,"15min":0.42799999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.52600000000000002,"5min":3.5649999999999999,"15min":6.4119999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56899999999999995,"15min":0.44400000000000001},"min":{"1min":0.24299999999999999,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":5.8179999999999996,"5min":5.8179999999999996,"15min":5.8179999999999996},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4242268160,"data_stored":4814725120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4092665856,"data_stored":4639543315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8330760192,"data_stored":9450094611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:21:21.593 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:21.594 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:41.595 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:21:42.009 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:21:42.009 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:21:42.034 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2388,"stamp":"2023-10-21T05:21:41.762740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28249910504,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2063587,"num_read_kb":8073425,"num_write":2044570,"num_write_kb":8177259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77859,"ondisk_log_size":77859,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22059552,"kb_used_data":16352388,"kb_used_omap":1771,"kb_used_meta":5705364,"kb_avail":261055968,"statfs":{"total":289910292480,"available":267321311232,"internally_reserved":0,"allocated":16744845312,"data_stored":18981732447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1814442,"internal_metadata":5842292822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,13,11,7,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18169856,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5028,"num_read_kb":20012,"num_write":4988,"num_write_kb":19952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002096"},"pg_stats":[{"pgid":"2.7","version":"88'10099","reported_seq":13904,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:39.063315+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:21:39.063315+0000","last_peered":"2023-10-21T05:21:39.063315+0000","last_clean":"2023-10-21T05:21:39.063315+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:21:39.063315+0000","last_undegraded":"2023-10-21T05:21:39.063315+0000","last_fullsized":"2023-10-21T05:21:39.063315+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1008,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:59.478791+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:20:59.478791+0000","last_peered":"2023-10-21T05:20:59.478791+0000","last_clean":"2023-10-21T05:20:59.478791+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:20:59.478791+0000","last_undegraded":"2023-10-21T05:20:59.478791+0000","last_fullsized":"2023-10-21T05:20:59.478791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"88'16666","reported_seq":22829,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:39.073442+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:21:39.073442+0000","last_peered":"2023-10-21T05:21:39.073442+0000","last_clean":"2023-10-21T05:21:39.073442+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:21:39.073442+0000","last_undegraded":"2023-10-21T05:21:39.073442+0000","last_fullsized":"2023-10-21T05:21:39.073442+0000","mapping_epoch":12,"log_start":"75'6600","ondisk_log_start":"75'6600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"88'3450","reported_seq":5320,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:34.947482+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:21:34.947482+0000","last_peered":"2023-10-21T05:21:34.947482+0000","last_clean":"2023-10-21T05:21:34.947482+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:21:34.947482+0000","last_undegraded":"2023-10-21T05:21:34.947482+0000","last_fullsized":"2023-10-21T05:21:34.947482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3450,"log_dups_size":0,"ondisk_log_size":3450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"88'517194","reported_seq":1030755,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:40.296032+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:21:40.296032+0000","last_peered":"2023-10-21T05:21:40.296032+0000","last_clean":"2023-10-21T05:21:40.296032+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:21:40.296032+0000","last_undegraded":"2023-10-21T05:21:40.296032+0000","last_fullsized":"2023-10-21T05:21:40.296032+0000","mapping_epoch":15,"log_start":"87'507117","ondisk_log_start":"87'507117","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7100510208,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":514878,"num_read_kb":2028724,"num_write":513436,"num_write_kb":2053744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"88'17099","reported_seq":23497,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:40.073946+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:21:40.073946+0000","last_peered":"2023-10-21T05:21:40.073946+0000","last_clean":"2023-10-21T05:21:40.073946+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:21:40.073946+0000","last_undegraded":"2023-10-21T05:21:40.073946+0000","last_fullsized":"2023-10-21T05:21:40.073946+0000","mapping_epoch":12,"log_start":"75'7000","ondisk_log_start":"75'7000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"88'510814","reported_seq":1016588,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:39.675864+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:21:39.675864+0000","last_peered":"2023-10-21T05:21:39.675864+0000","last_clean":"2023-10-21T05:21:39.675864+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:21:39.675864+0000","last_undegraded":"2023-10-21T05:21:39.675864+0000","last_fullsized":"2023-10-21T05:21:39.675864+0000","mapping_epoch":15,"log_start":"87'500719","ondisk_log_start":"87'500719","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":1863,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 273s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":7003082752,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":507014,"num_read_kb":1997040,"num_write":507901,"num_write_kb":2031604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6650,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:34.949352+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:21:34.949352+0000","last_peered":"2023-10-21T05:21:34.949352+0000","last_clean":"2023-10-21T05:21:34.949352+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:21:34.949352+0000","last_undegraded":"2023-10-21T05:21:34.949352+0000","last_fullsized":"2023-10-21T05:21:34.949352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"88'531209","reported_seq":1057226,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:40.293968+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:21:40.293968+0000","last_peered":"2023-10-21T05:21:40.293968+0000","last_clean":"2023-10-21T05:21:40.293968+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:21:40.293968+0000","last_undegraded":"2023-10-21T05:21:40.293968+0000","last_fullsized":"2023-10-21T05:21:40.293968+0000","mapping_epoch":15,"log_start":"87'521125","ondisk_log_start":"87'521125","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7301115904,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":527386,"num_read_kb":2077940,"num_write":527333,"num_write_kb":2109332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":167,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.554853+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:21:00.554853+0000","last_peered":"2023-10-21T05:21:00.554853+0000","last_clean":"2023-10-21T05:21:00.554853+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:21:00.554853+0000","last_undegraded":"2023-10-21T05:21:00.554853+0000","last_fullsized":"2023-10-21T05:21:00.554853+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"88'498725","reported_seq":993608,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:39.678597+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:21:39.678597+0000","last_peered":"2023-10-21T05:21:39.678597+0000","last_clean":"2023-10-21T05:21:39.678597+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:21:39.678597+0000","last_undegraded":"2023-10-21T05:21:39.678597+0000","last_fullsized":"2023-10-21T05:21:39.678597+0000","mapping_epoch":15,"log_start":"87'488718","ondisk_log_start":"87'488718","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6844608531,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":496144,"num_read_kb":1954244,"num_write":495094,"num_write_kb":1980370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'3431","reported_seq":4837,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:34.947411+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:21:34.947411+0000","last_peered":"2023-10-21T05:21:34.947411+0000","last_clean":"2023-10-21T05:21:34.947411+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:21:34.947411+0000","last_undegraded":"2023-10-21T05:21:34.947411+0000","last_fullsized":"2023-10-21T05:21:34.947411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3431,"log_dups_size":0,"ondisk_log_size":3431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":264,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.541199+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:21:00.541199+0000","last_peered":"2023-10-21T05:21:00.541199+0000","last_clean":"2023-10-21T05:21:00.541199+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:21:00.541199+0000","last_undegraded":"2023-10-21T05:21:00.541199+0000","last_fullsized":"2023-10-21T05:21:00.541199+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28249317395,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2045422,"num_read_kb":8057948,"num_write":2043764,"num_write_kb":8175050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16742535168,"data_stored":18981204006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40263,"ondisk_log_size":40263,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37497,"ondisk_log_size":37497,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739293,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6104064,"kb_used_data":4164472,"kb_used_omap":373,"kb_used_meta":1939210,"kb_avail":88267776,"statfs":{"total":96636764160,"available":90386202624,"internally_reserved":0,"allocated":4264419328,"data_stored":4836274437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":382416,"internal_metadata":1985751600},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,9,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:21:14 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.46700000000000003,"15min":0.39700000000000002},"min":{"1min":0.161,"5min":0.161,"15min":0.14299999999999999},"max":{"1min":0.72699999999999998,"5min":4.0259999999999998,"15min":4.9889999999999999},"last":0.755},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56699999999999995,"15min":0.47499999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.182},"max":{"1min":5.923,"5min":6.992,"15min":6.992},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Oct 21 05:21:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.61699999999999999,"15min":0.55800000000000005},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":3.4420000000000002,"5min":5.4390000000000001,"15min":14.318},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.65900000000000003,"15min":0.60199999999999998},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.245},"max":{"1min":3.464,"5min":5.4740000000000002,"15min":14.343},"last":0.46200000000000002}]}]},{"osd":2,"up_from":8,"seq":34359739292,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9975364,"kb_used_data":8174844,"kb_used_omap":1038,"kb_used_meta":1799473,"kb_avail":84396476,"statfs":{"total":96636764160,"available":86421991424,"internally_reserved":0,"allocated":8371040256,"data_stored":9489881837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1063420,"internal_metadata":1842660868},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:03 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.438,"15min":0.48199999999999998},"min":{"1min":0.19400000000000001,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":1.03,"5min":7.234,"15min":7.234},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.40000000000000002,"15min":0.34999999999999998},"min":{"1min":0.214,"5min":0.191,"15min":0.191},"max":{"1min":4.4420000000000002,"5min":4.4420000000000002,"15min":4.4420000000000002},"last":0.33300000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:21:03 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.35399999999999998,"15min":0.40100000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.21199999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":4.0510000000000002},"last":0.311},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.38600000000000001,"15min":0.41699999999999998},"min":{"1min":0.28199999999999997,"5min":0.22800000000000001,"15min":0.20000000000000001},"max":{"1min":0.91500000000000004,"5min":2.722,"15min":3.8370000000000002},"last":0.23899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739291,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5980124,"kb_used_data":4013072,"kb_used_omap":359,"kb_used_meta":1966680,"kb_avail":88391716,"statfs":{"total":96636764160,"available":90513117184,"internally_reserved":0,"allocated":4109385728,"data_stored":4655576173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":2013880354},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,9,7,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45100000000000001,"15min":0.40300000000000002},"min":{"1min":0.17199999999999999,"5min":0.161,"15min":0.161},"max":{"1min":2.8690000000000002,"5min":5.0490000000000004,"15min":5.0490000000000004},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.39300000000000002,"15min":0.45400000000000001},"min":{"1min":0.19500000000000001,"5min":0.19,"15min":0.17000000000000001},"max":{"1min":2.8359999999999999,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.251}]},{"osd":2,"last update":"Sat Oct 21 05:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":3.7280000000000002,"5min":3.7280000000000002,"15min":6.4119999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.53400000000000003,"15min":0.443},"min":{"1min":0.26700000000000002,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":0.38700000000000001,"5min":5.8179999999999996,"15min":5.8179999999999996},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4262989824,"data_stored":4835446784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4108263424,"data_stored":4655140883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8371281920,"data_stored":9490616339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:21:42.036 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:42.037 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:56.434 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:21:56.428+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:21:56.444 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:21:56.440+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:21:56.445 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:21:56.440+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:21:56.445 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:21:56.440+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:22:02.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:22:02.472 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:22:02.473 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:22:02.514 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2398,"stamp":"2023-10-21T05:22:01.765922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28278422760,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2071309,"num_read_kb":8104153,"num_write":2052375,"num_write_kb":8208479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77649,"ondisk_log_size":77649,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21763380,"kb_used_data":16414876,"kb_used_omap":1773,"kb_used_meta":5346706,"kb_avail":261352140,"statfs":{"total":289910292480,"available":267624591360,"internally_reserved":0,"allocated":16808833024,"data_stored":19045720159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1816122,"internal_metadata":5475027398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,6,4,7,6,8,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18530304,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5044,"num_read_kb":20060,"num_write":5079,"num_write_kb":20316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001945"},"pg_stats":[{"pgid":"2.7","version":"88'10155","reported_seq":13972,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:58.145635+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:21:58.145635+0000","last_peered":"2023-10-21T05:21:58.145635+0000","last_clean":"2023-10-21T05:21:58.145635+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:21:58.145635+0000","last_undegraded":"2023-10-21T05:21:58.145635+0000","last_fullsized":"2023-10-21T05:21:58.145635+0000","mapping_epoch":12,"log_start":"39'100","ondisk_log_start":"39'100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1008,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:20:59.478791+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:20:59.478791+0000","last_peered":"2023-10-21T05:20:59.478791+0000","last_clean":"2023-10-21T05:20:59.478791+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:20:59.478791+0000","last_undegraded":"2023-10-21T05:20:59.478791+0000","last_fullsized":"2023-10-21T05:20:59.478791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"88'16757","reported_seq":22940,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:58.159064+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:21:58.159064+0000","last_peered":"2023-10-21T05:21:58.159064+0000","last_clean":"2023-10-21T05:21:58.159064+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:21:58.159064+0000","last_undegraded":"2023-10-21T05:21:58.159064+0000","last_fullsized":"2023-10-21T05:21:58.159064+0000","mapping_epoch":12,"log_start":"75'6700","ondisk_log_start":"75'6700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"88'3469","reported_seq":5343,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:57.412639+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:21:57.412639+0000","last_peered":"2023-10-21T05:21:57.412639+0000","last_clean":"2023-10-21T05:21:57.412639+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:21:57.412639+0000","last_undegraded":"2023-10-21T05:21:57.412639+0000","last_fullsized":"2023-10-21T05:21:57.412639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"88'519125","reported_seq":1034588,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:22:00.330973+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:22:00.330973+0000","last_peered":"2023-10-21T05:22:00.330973+0000","last_clean":"2023-10-21T05:22:00.330973+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:22:00.330973+0000","last_undegraded":"2023-10-21T05:22:00.330973+0000","last_fullsized":"2023-10-21T05:22:00.330973+0000","mapping_epoch":15,"log_start":"87'509117","ondisk_log_start":"87'509117","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7107596288,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":516788,"num_read_kb":2036300,"num_write":515367,"num_write_kb":2061468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"88'17169","reported_seq":23582,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:22:00.044227+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:22:00.044227+0000","last_peered":"2023-10-21T05:22:00.044227+0000","last_clean":"2023-10-21T05:22:00.044227+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:22:00.044227+0000","last_undegraded":"2023-10-21T05:22:00.044227+0000","last_fullsized":"2023-10-21T05:22:00.044227+0000","mapping_epoch":12,"log_start":"75'7100","ondisk_log_start":"75'7100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"88'512868","reported_seq":1020570,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:59.724504+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:21:59.724504+0000","last_peered":"2023-10-21T05:21:59.724504+0000","last_clean":"2023-10-21T05:21:59.724504+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:21:59.724504+0000","last_undegraded":"2023-10-21T05:21:59.724504+0000","last_fullsized":"2023-10-21T05:21:59.724504+0000","mapping_epoch":15,"log_start":"87'502819","ondisk_log_start":"87'502819","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2070,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 293s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":7010013184,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":508922,"num_read_kb":2004644,"num_write":509825,"num_write_kb":2039300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6670,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:55.432314+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:21:55.432314+0000","last_peered":"2023-10-21T05:21:55.432314+0000","last_clean":"2023-10-21T05:21:55.432314+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:21:55.432314+0000","last_undegraded":"2023-10-21T05:21:55.432314+0000","last_fullsized":"2023-10-21T05:21:55.432314+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"88'533260","reported_seq":1061297,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:22:00.330774+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:22:00.330774+0000","last_peered":"2023-10-21T05:22:00.330774+0000","last_clean":"2023-10-21T05:22:00.330774+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:22:00.330774+0000","last_undegraded":"2023-10-21T05:22:00.330774+0000","last_fullsized":"2023-10-21T05:22:00.330774+0000","mapping_epoch":15,"log_start":"87'523225","ondisk_log_start":"87'523225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7308668928,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":529406,"num_read_kb":2085980,"num_write":529384,"num_write_kb":2117536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":167,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.554853+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:21:00.554853+0000","last_peered":"2023-10-21T05:21:00.554853+0000","last_clean":"2023-10-21T05:21:00.554853+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:21:00.554853+0000","last_undegraded":"2023-10-21T05:21:00.554853+0000","last_fullsized":"2023-10-21T05:21:00.554853+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"88'500624","reported_seq":997393,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:59.714544+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:21:59.714544+0000","last_peered":"2023-10-21T05:21:59.714544+0000","last_clean":"2023-10-21T05:21:59.714544+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:21:59.714544+0000","last_undegraded":"2023-10-21T05:21:59.714544+0000","last_fullsized":"2023-10-21T05:21:59.714544+0000","mapping_epoch":15,"log_start":"87'490618","ondisk_log_start":"87'490618","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6851551251,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":498028,"num_read_kb":1961752,"num_write":496993,"num_write_kb":1987966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'3450","reported_seq":4860,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:57.438323+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:21:57.438323+0000","last_peered":"2023-10-21T05:21:57.438323+0000","last_clean":"2023-10-21T05:21:57.438323+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:21:57.438323+0000","last_undegraded":"2023-10-21T05:21:57.438323+0000","last_fullsized":"2023-10-21T05:21:57.438323+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3450,"log_dups_size":0,"ondisk_log_size":3450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":264,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.541199+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:21:00.541199+0000","last_peered":"2023-10-21T05:21:00.541199+0000","last_clean":"2023-10-21T05:21:00.541199+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:21:00.541199+0000","last_undegraded":"2023-10-21T05:21:00.541199+0000","last_fullsized":"2023-10-21T05:21:00.541199+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28277829651,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2053144,"num_read_kb":8088676,"num_write":2051569,"num_write_kb":8206270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16806457344,"data_stored":19045126182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40098,"ondisk_log_size":40098,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37452,"ondisk_log_size":37452,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632744,"kb_used_data":4180384,"kb_used_omap":375,"kb_used_meta":1451976,"kb_avail":88739096,"statfs":{"total":96636764160,"available":90868834304,"internally_reserved":0,"allocated":4280713216,"data_stored":4852568325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":384096,"internal_metadata":1486824352},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:21:14 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.46700000000000003,"15min":0.39700000000000002},"min":{"1min":0.161,"5min":0.161,"15min":0.14299999999999999},"max":{"1min":0.72699999999999998,"5min":4.0259999999999998,"15min":4.9889999999999999},"last":0.16500000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56699999999999995,"15min":0.47499999999999998},"min":{"1min":0.214,"5min":0.182,"15min":0.182},"max":{"1min":5.923,"5min":6.992,"15min":6.992},"last":0.222}]},{"osd":2,"last update":"Sat Oct 21 05:21:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.61699999999999999,"15min":0.55800000000000005},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.19900000000000001},"max":{"1min":3.4420000000000002,"5min":5.4390000000000001,"15min":14.318},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.65900000000000003,"15min":0.60199999999999998},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.245},"max":{"1min":3.464,"5min":5.4740000000000002,"15min":14.343},"last":0.27000000000000002}]}]},{"osd":2,"up_from":8,"seq":34359739296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10074796,"kb_used_data":8206244,"kb_used_omap":1038,"kb_used_meta":1867505,"kb_avail":84297044,"statfs":{"total":96636764160,"available":86320173056,"internally_reserved":0,"allocated":8403193856,"data_stored":9522035437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1063420,"internal_metadata":1912325636},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:03 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.438,"15min":0.48199999999999998},"min":{"1min":0.19400000000000001,"5min":0.157,"15min":0.14599999999999999},"max":{"1min":1.03,"5min":7.234,"15min":7.234},"last":0.378},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.40000000000000002,"15min":0.34999999999999998},"min":{"1min":0.214,"5min":0.191,"15min":0.191},"max":{"1min":4.4420000000000002,"5min":4.4420000000000002,"15min":4.4420000000000002},"last":0.26600000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:21:03 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.35399999999999998,"15min":0.40100000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.21199999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":4.0510000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.38600000000000001,"15min":0.41699999999999998},"min":{"1min":0.28199999999999997,"5min":0.22800000000000001,"15min":0.20000000000000001},"max":{"1min":0.91500000000000004,"5min":2.722,"15min":3.8370000000000002},"last":0.311}]}]},{"osd":1,"up_from":8,"seq":34359739295,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6055840,"kb_used_data":4028248,"kb_used_omap":359,"kb_used_meta":2027224,"kb_avail":88316000,"statfs":{"total":96636764160,"available":90435584000,"internally_reserved":0,"allocated":4124925952,"data_stored":4671116397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":2075877410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,1,6,6,8,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45100000000000001,"15min":0.40300000000000002},"min":{"1min":0.17199999999999999,"5min":0.161,"15min":0.161},"max":{"1min":2.8690000000000002,"5min":5.0490000000000004,"15min":5.0490000000000004},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.39300000000000002,"15min":0.45400000000000001},"min":{"1min":0.19500000000000001,"5min":0.19,"15min":0.17000000000000001},"max":{"1min":2.8359999999999999,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.22700000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":3.7280000000000002,"5min":3.7280000000000002,"15min":6.4119999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.53400000000000003,"15min":0.443},"min":{"1min":0.26700000000000002,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":0.38700000000000001,"5min":5.8179999999999996,"15min":5.8179999999999996},"last":0.308}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4279308288,"data_stored":4851765248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4123930624,"data_stored":4670808083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8403218432,"data_stored":9522552851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:22:02.516 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:22:02.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:22:03.017 INFO:teuthology.orchestra.run.smithi119.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:22:03.045 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:22:13.519 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:13.508+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:22:13.519 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:13.508+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:22:13.520 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:13.508+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:22:16.643 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:16.635+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:22:16.654 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:16.647+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:22:16.663 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:16.659+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:22:16.664 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:16.659+0000 7f874a7e7640 -1 reset not still connected to 0x557670fce820 2023-10-21T05:22:16.664 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:16.659+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:22:16.664 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:16.659+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:22:16.665 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:16.659+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:22:16.665 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:16.659+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:22:23.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:22:23.549 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:22:23.550 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:22:23.586 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2409,"stamp":"2023-10-21T05:22:21.769221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28308303080,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2079291,"num_read_kb":8135841,"num_write":2060593,"num_write_kb":8241351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77682,"ondisk_log_size":77682,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21986428,"kb_used_data":16481060,"kb_used_omap":1773,"kb_used_meta":5503570,"kb_avail":261129092,"statfs":{"total":289910292480,"available":267396190208,"internally_reserved":0,"allocated":16876605440,"data_stored":19113478598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1816122,"internal_metadata":5635656134},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,6,8,9,19,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15241216,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4066,"num_read_kb":16152,"num_write":4191,"num_write_kb":16764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001593"},"pg_stats":[{"pgid":"2.7","version":"89'10210","reported_seq":14040,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:18.878453+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:22:18.878453+0000","last_peered":"2023-10-21T05:22:18.878453+0000","last_clean":"2023-10-21T05:22:18.878453+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:22:18.878453+0000","last_undegraded":"2023-10-21T05:22:18.878453+0000","last_fullsized":"2023-10-21T05:22:18.878453+0000","mapping_epoch":12,"log_start":"41'200","ondisk_log_start":"41'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1009,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.540893+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:21:00.540893+0000","last_peered":"2023-10-21T05:21:00.540893+0000","last_clean":"2023-10-21T05:21:00.540893+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:21:00.540893+0000","last_undegraded":"2023-10-21T05:21:00.540893+0000","last_fullsized":"2023-10-21T05:21:00.540893+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'16847","reported_seq":23051,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:18.886869+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:22:18.886869+0000","last_peered":"2023-10-21T05:22:18.886869+0000","last_clean":"2023-10-21T05:22:18.886869+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:22:18.886869+0000","last_undegraded":"2023-10-21T05:22:18.886869+0000","last_fullsized":"2023-10-21T05:22:18.886869+0000","mapping_epoch":12,"log_start":"75'6800","ondisk_log_start":"75'6800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'3488","reported_seq":5367,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:16.637400+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:22:16.637400+0000","last_peered":"2023-10-21T05:22:16.637400+0000","last_clean":"2023-10-21T05:22:16.637400+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:22:16.637400+0000","last_undegraded":"2023-10-21T05:22:16.637400+0000","last_fullsized":"2023-10-21T05:22:16.637400+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3488,"log_dups_size":0,"ondisk_log_size":3488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'521197","reported_seq":1038652,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:20.372925+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:22:20.372925+0000","last_peered":"2023-10-21T05:22:20.372925+0000","last_clean":"2023-10-21T05:22:20.372925+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:22:20.372925+0000","last_undegraded":"2023-10-21T05:22:20.372925+0000","last_fullsized":"2023-10-21T05:22:20.372925+0000","mapping_epoch":15,"log_start":"87'511117","ondisk_log_start":"87'511117","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7115128832,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":518771,"num_read_kb":2044172,"num_write":517439,"num_write_kb":2069756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'17244","reported_seq":23678,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:18.673945+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:22:18.673945+0000","last_peered":"2023-10-21T05:22:18.673945+0000","last_clean":"2023-10-21T05:22:18.673945+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:22:18.673945+0000","last_undegraded":"2023-10-21T05:22:18.673945+0000","last_fullsized":"2023-10-21T05:22:18.673945+0000","mapping_epoch":12,"log_start":"75'7200","ondisk_log_start":"75'7200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'514952","reported_seq":1024607,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:19.838512+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:22:19.838512+0000","last_peered":"2023-10-21T05:22:19.838512+0000","last_clean":"2023-10-21T05:22:19.838512+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:22:19.838512+0000","last_undegraded":"2023-10-21T05:22:19.838512+0000","last_fullsized":"2023-10-21T05:22:19.838512+0000","mapping_epoch":15,"log_start":"87'504920","ondisk_log_start":"87'504920","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2168,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 313s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":7017361408,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":510865,"num_read_kb":2012360,"num_write":511851,"num_write_kb":2047404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6692,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.912168+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:22:15.912168+0000","last_peered":"2023-10-21T05:22:15.912168+0000","last_clean":"2023-10-21T05:22:15.912168+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:22:15.912168+0000","last_undegraded":"2023-10-21T05:22:15.912168+0000","last_fullsized":"2023-10-21T05:22:15.912168+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'535329","reported_seq":1065397,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:20.365304+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:22:20.365304+0000","last_peered":"2023-10-21T05:22:20.365304+0000","last_clean":"2023-10-21T05:22:20.365304+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:22:20.365304+0000","last_undegraded":"2023-10-21T05:22:20.365304+0000","last_fullsized":"2023-10-21T05:22:20.365304+0000","mapping_epoch":15,"log_start":"87'525325","ondisk_log_start":"87'525325","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7316172800,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":531437,"num_read_kb":2094032,"num_write":531453,"num_write_kb":2125812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":168,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:29.952539+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:21:29.952539+0000","last_peered":"2023-10-21T05:21:29.952539+0000","last_clean":"2023-10-21T05:21:29.952539+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:21:29.952539+0000","last_undegraded":"2023-10-21T05:21:29.952539+0000","last_fullsized":"2023-10-21T05:21:29.952539+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'502675","reported_seq":1001468,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:19.818357+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:22:19.818357+0000","last_peered":"2023-10-21T05:22:19.818357+0000","last_clean":"2023-10-21T05:22:19.818357+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:22:19.818357+0000","last_undegraded":"2023-10-21T05:22:19.818357+0000","last_fullsized":"2023-10-21T05:22:19.818357+0000","mapping_epoch":15,"log_start":"87'492618","ondisk_log_start":"87'492618","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6859046931,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":500053,"num_read_kb":1969800,"num_write":499044,"num_write_kb":1996170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'3469","reported_seq":4884,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:16.657600+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:22:16.657600+0000","last_peered":"2023-10-21T05:22:16.657600+0000","last_clean":"2023-10-21T05:22:16.657600+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:22:16.657600+0000","last_undegraded":"2023-10-21T05:22:16.657600+0000","last_fullsized":"2023-10-21T05:22:16.657600+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3469,"log_dups_size":0,"ondisk_log_size":3469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":265,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:24.583018+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:21:24.583018+0000","last_peered":"2023-10-21T05:21:24.583018+0000","last_clean":"2023-10-21T05:21:24.583018+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:21:24.583018+0000","last_undegraded":"2023-10-21T05:21:24.583018+0000","last_fullsized":"2023-10-21T05:21:24.583018+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28307709971,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2061126,"num_read_kb":8120364,"num_write":2059787,"num_write_kb":8239142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16873762816,"data_stored":19112431654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40173,"ondisk_log_size":40173,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37410,"ondisk_log_size":37410,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739301,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5667256,"kb_used_data":4196848,"kb_used_omap":375,"kb_used_meta":1470024,"kb_avail":88704584,"statfs":{"total":96636764160,"available":90833494016,"internally_reserved":0,"allocated":4297572352,"data_stored":4869422802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":384096,"internal_metadata":1505305504},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.25800000000000001,"5min":0.40699999999999997,"15min":0.38900000000000001},"min":{"1min":0.16500000000000001,"5min":0.161,"15min":0.14599999999999999},"max":{"1min":0.755,"5min":3.5329999999999999,"15min":4.9889999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.61799999999999999,"15min":0.51400000000000001},"min":{"1min":0.222,"5min":0.182,"15min":0.182},"max":{"1min":4.6429999999999998,"5min":6.992,"15min":6.992},"last":0.251}]},{"osd":2,"last update":"Sat Oct 21 05:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57199999999999995,"15min":0.504},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":5.3479999999999999,"5min":5.4390000000000001,"15min":5.4390000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61599999999999999,"15min":0.54800000000000004},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":5.407,"5min":5.4740000000000002,"15min":5.4740000000000002},"last":0.36099999999999999}]}]},{"osd":2,"up_from":8,"seq":34359739300,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10210448,"kb_used_data":8239176,"kb_used_omap":1038,"kb_used_meta":1970225,"kb_avail":84161392,"statfs":{"total":96636764160,"available":86181265408,"internally_reserved":0,"allocated":8436916224,"data_stored":9555753146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1063420,"internal_metadata":2017510916},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.43099999999999999,"15min":0.47899999999999998},"min":{"1min":0.17599999999999999,"5min":0.16900000000000001,"15min":0.157},"max":{"1min":2.7610000000000001,"5min":7.234,"15min":7.234},"last":0.372},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42199999999999999,"15min":0.34999999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3380000000000001,"5min":4.4420000000000002,"15min":4.4420000000000002},"last":0.51100000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.38900000000000001,"15min":0.41499999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.335,"5min":4.335,"15min":4.335},"last":0.246},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.42599999999999999,"15min":0.42799999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20000000000000001},"max":{"1min":4.2839999999999998,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.26700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739300,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108724,"kb_used_data":4045036,"kb_used_omap":359,"kb_used_meta":2063320,"kb_avail":88263116,"statfs":{"total":96636764160,"available":90381430784,"internally_reserved":0,"allocated":4142116864,"data_stored":4688302650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":368606,"internal_metadata":2112839714},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,5,2,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45100000000000001,"15min":0.40300000000000002},"min":{"1min":0.17199999999999999,"5min":0.161,"15min":0.161},"max":{"1min":2.8690000000000002,"5min":5.0490000000000004,"15min":5.0490000000000004},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.39300000000000002,"15min":0.45400000000000001},"min":{"1min":0.19500000000000001,"5min":0.19,"15min":0.17000000000000001},"max":{"1min":2.8359999999999999,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.372}]},{"osd":2,"last update":"Sat Oct 21 05:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":3.7280000000000002,"5min":3.7280000000000002,"15min":6.4119999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.53400000000000003,"15min":0.443},"min":{"1min":0.26700000000000002,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":0.38700000000000001,"5min":5.8179999999999996,"15min":5.8179999999999996},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4296261632,"data_stored":4868718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4140630016,"data_stored":4687507475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8436871168,"data_stored":9556205587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:22:23.588 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:22:23.588 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:22:26.296 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:26.291+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:22:26.310 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:26.303+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:22:26.311 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:26.303+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:22:26.311 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:26.303+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:22:38.244 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:38.223+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:22:38.255 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:38.239+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:22:38.255 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:38.239+0000 7f874a7e7640 -1 reset not still connected to 0x557670489520 2023-10-21T05:22:38.255 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:38.239+0000 7f874a7e7640 -1 reset not still connected to 0x557670a6fd40 2023-10-21T05:22:38.256 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:38.239+0000 7f874a7e7640 -1 reset not still connected to 0x55767136e680 2023-10-21T05:22:38.256 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:38.239+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:22:38.256 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:38.239+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:22:38.256 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:38.239+0000 7f874a7e7640 -1 reset not still connected to 0x557671c1fe10 2023-10-21T05:22:43.590 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:22:44.203 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:22:44.204 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:22:44.237 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2420,"stamp":"2023-10-21T05:22:43.773014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28337204456,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2087434,"num_read_kb":8168265,"num_write":2068503,"num_write_kb":8272991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77809,"ondisk_log_size":77809,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21693576,"kb_used_data":16543148,"kb_used_omap":1775,"kb_used_meta":5148624,"kb_avail":261421944,"statfs":{"total":289910292480,"available":267696070656,"internally_reserved":0,"allocated":16940183552,"data_stored":19177056710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1817801,"internal_metadata":5272191799},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,2,4,8,4,6,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14753792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4082,"num_read_kb":16260,"num_write":4028,"num_write_kb":16112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002020"},"pg_stats":[{"pgid":"2.7","version":"89'10251","reported_seq":14093,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:38.448633+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:22:38.448633+0000","last_peered":"2023-10-21T05:22:38.448633+0000","last_clean":"2023-10-21T05:22:38.448633+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:22:38.448633+0000","last_undegraded":"2023-10-21T05:22:38.448633+0000","last_fullsized":"2023-10-21T05:22:38.448633+0000","mapping_epoch":12,"log_start":"41'200","ondisk_log_start":"41'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1009,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.540893+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:21:00.540893+0000","last_peered":"2023-10-21T05:21:00.540893+0000","last_clean":"2023-10-21T05:21:00.540893+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:21:00.540893+0000","last_undegraded":"2023-10-21T05:21:00.540893+0000","last_fullsized":"2023-10-21T05:21:00.540893+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'16914","reported_seq":23139,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:38.463668+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:22:38.463668+0000","last_peered":"2023-10-21T05:22:38.463668+0000","last_clean":"2023-10-21T05:22:38.463668+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:22:38.463668+0000","last_undegraded":"2023-10-21T05:22:38.463668+0000","last_fullsized":"2023-10-21T05:22:38.463668+0000","mapping_epoch":12,"log_start":"75'6900","ondisk_log_start":"75'6900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'3503","reported_seq":5387,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:38.232316+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:22:38.232316+0000","last_peered":"2023-10-21T05:22:38.232316+0000","last_clean":"2023-10-21T05:22:38.232316+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:22:38.232316+0000","last_undegraded":"2023-10-21T05:22:38.232316+0000","last_fullsized":"2023-10-21T05:22:38.232316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'523169","reported_seq":1042578,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:40.429413+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:22:40.429413+0000","last_peered":"2023-10-21T05:22:40.429413+0000","last_clean":"2023-10-21T05:22:40.429413+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:22:40.429413+0000","last_undegraded":"2023-10-21T05:22:40.429413+0000","last_fullsized":"2023-10-21T05:22:40.429413+0000","mapping_epoch":15,"log_start":"87'513117","ondisk_log_start":"87'513117","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7122313216,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":520725,"num_read_kb":2051936,"num_write":519411,"num_write_kb":2077644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'17329","reported_seq":23784,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:39.989368+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:22:39.989368+0000","last_peered":"2023-10-21T05:22:39.989368+0000","last_clean":"2023-10-21T05:22:39.989368+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:22:39.989368+0000","last_undegraded":"2023-10-21T05:22:39.989368+0000","last_fullsized":"2023-10-21T05:22:39.989368+0000","mapping_epoch":12,"log_start":"75'7300","ondisk_log_start":"75'7300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'517067","reported_seq":1028832,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:39.948471+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:22:39.948471+0000","last_peered":"2023-10-21T05:22:39.948471+0000","last_clean":"2023-10-21T05:22:39.948471+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:22:39.948471+0000","last_undegraded":"2023-10-21T05:22:39.948471+0000","last_fullsized":"2023-10-21T05:22:39.948471+0000","mapping_epoch":15,"log_start":"88'507020","ondisk_log_start":"88'507020","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2326,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 333s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":7024635904,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":512953,"num_read_kb":2020692,"num_write":513872,"num_write_kb":2055488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6712,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:36.401088+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:22:36.401088+0000","last_peered":"2023-10-21T05:22:36.401088+0000","last_clean":"2023-10-21T05:22:36.401088+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:22:36.401088+0000","last_undegraded":"2023-10-21T05:22:36.401088+0000","last_fullsized":"2023-10-21T05:22:36.401088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'537305","reported_seq":1069511,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:40.426222+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:22:40.426222+0000","last_peered":"2023-10-21T05:22:40.426222+0000","last_clean":"2023-10-21T05:22:40.426222+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:22:40.426222+0000","last_undegraded":"2023-10-21T05:22:40.426222+0000","last_fullsized":"2023-10-21T05:22:40.426222+0000","mapping_epoch":15,"log_start":"88'527225","ondisk_log_start":"88'527225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7323512832,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":533576,"num_read_kb":2102552,"num_write":533429,"num_write_kb":2133716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":168,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:29.952539+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:21:29.952539+0000","last_peered":"2023-10-21T05:21:29.952539+0000","last_clean":"2023-10-21T05:21:29.952539+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:21:29.952539+0000","last_undegraded":"2023-10-21T05:21:29.952539+0000","last_fullsized":"2023-10-21T05:21:29.952539+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'504616","reported_seq":1005373,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:39.940241+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:22:39.940241+0000","last_peered":"2023-10-21T05:22:39.940241+0000","last_clean":"2023-10-21T05:22:39.940241+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:22:39.940241+0000","last_undegraded":"2023-10-21T05:22:39.940241+0000","last_fullsized":"2023-10-21T05:22:39.940241+0000","mapping_epoch":15,"log_start":"87'494518","ondisk_log_start":"87'494518","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6866149395,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":502015,"num_read_kb":1977608,"num_write":500985,"num_write_kb":2003934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'3484","reported_seq":4904,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:38.225237+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:22:38.225237+0000","last_peered":"2023-10-21T05:22:38.225237+0000","last_clean":"2023-10-21T05:22:38.225237+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:22:38.225237+0000","last_undegraded":"2023-10-21T05:22:38.225237+0000","last_fullsized":"2023-10-21T05:22:38.225237+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3484,"log_dups_size":0,"ondisk_log_size":3484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":265,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:24.583018+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:21:24.583018+0000","last_peered":"2023-10-21T05:21:24.583018+0000","last_clean":"2023-10-21T05:21:24.583018+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:21:24.583018+0000","last_undegraded":"2023-10-21T05:21:24.583018+0000","last_fullsized":"2023-10-21T05:21:24.583018+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28336611347,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2069269,"num_read_kb":8152788,"num_write":2067697,"num_write_kb":8270782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16938520576,"data_stored":19177189414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40277,"ondisk_log_size":40277,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37433,"ondisk_log_size":37433,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739305,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5718944,"kb_used_data":4212440,"kb_used_omap":375,"kb_used_meta":1506120,"kb_avail":88652896,"statfs":{"total":96636764160,"available":90780565504,"internally_reserved":0,"allocated":4313538560,"data_stored":4885389010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":384096,"internal_metadata":1542267808},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.25800000000000001,"5min":0.40699999999999997,"15min":0.38900000000000001},"min":{"1min":0.16500000000000001,"5min":0.161,"15min":0.14599999999999999},"max":{"1min":0.755,"5min":3.5329999999999999,"15min":4.9889999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.61799999999999999,"15min":0.51400000000000001},"min":{"1min":0.222,"5min":0.182,"15min":0.182},"max":{"1min":4.6429999999999998,"5min":6.992,"15min":6.992},"last":0.24099999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57199999999999995,"15min":0.504},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":5.3479999999999999,"5min":5.4390000000000001,"15min":5.4390000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61599999999999999,"15min":0.54800000000000004},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":5.407,"5min":5.4740000000000002,"15min":5.4740000000000002},"last":0.46800000000000003}]}]},{"osd":2,"up_from":8,"seq":34359739304,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10337712,"kb_used_data":8270500,"kb_used_omap":1038,"kb_used_meta":2066161,"kb_avail":84034128,"statfs":{"total":96636764160,"available":86050947072,"internally_reserved":0,"allocated":8468992000,"data_stored":9587828922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1063420,"internal_metadata":2115749380},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.43099999999999999,"15min":0.47899999999999998},"min":{"1min":0.17599999999999999,"5min":0.16900000000000001,"15min":0.157},"max":{"1min":2.7610000000000001,"5min":7.234,"15min":7.234},"last":0.184},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42199999999999999,"15min":0.34999999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3380000000000001,"5min":4.4420000000000002,"15min":4.4420000000000002},"last":0.46400000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.38900000000000001,"15min":0.41499999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.335,"5min":4.335,"15min":4.335},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.42599999999999999,"15min":0.42799999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20000000000000001},"max":{"1min":4.2839999999999998,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.41999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739304,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5636920,"kb_used_data":4060208,"kb_used_omap":361,"kb_used_meta":1576342,"kb_avail":88734920,"statfs":{"total":96636764160,"available":90864558080,"internally_reserved":0,"allocated":4157652992,"data_stored":4703838778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":370285,"internal_metadata":1614174611},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,2,3,2,6,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45100000000000001,"15min":0.40300000000000002},"min":{"1min":0.17199999999999999,"5min":0.161,"15min":0.161},"max":{"1min":2.8690000000000002,"5min":5.0490000000000004,"15min":5.0490000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.39300000000000002,"15min":0.45400000000000001},"min":{"1min":0.19500000000000001,"5min":0.19,"15min":0.17000000000000001},"max":{"1min":2.8359999999999999,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":3.7280000000000002,"5min":3.7280000000000002,"15min":6.4119999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.53400000000000003,"15min":0.443},"min":{"1min":0.26700000000000002,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":0.38700000000000001,"5min":5.8179999999999996,"15min":5.8179999999999996},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4312432640,"data_stored":4884889600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4156850176,"data_stored":4703727635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8469237760,"data_stored":9588572179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:22:44.239 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:22:44.240 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:22:59.322 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:59.306+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:22:59.324 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:22:59.310+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:23:04.241 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:23:04.743 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:23:04.744 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:23:04.791 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2430,"stamp":"2023-10-21T05:23:03.776525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28366056680,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2095286,"num_read_kb":8199513,"num_write":2076444,"num_write_kb":8304755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77983,"ondisk_log_size":77983,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21012852,"kb_used_data":16606492,"kb_used_omap":1776,"kb_used_meta":4404559,"kb_avail":262102668,"statfs":{"total":289910292480,"available":268393132032,"internally_reserved":0,"allocated":17005047808,"data_stored":19241920966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1819481,"internal_metadata":4510268583},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,4,16,5,9,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13918208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3769,"num_read_kb":14980,"num_write":3849,"num_write_kb":15396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002034"},"pg_stats":[{"pgid":"2.7","version":"89'10300","reported_seq":14154,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:59.435020+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:22:59.435020+0000","last_peered":"2023-10-21T05:22:59.435020+0000","last_clean":"2023-10-21T05:22:59.435020+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:22:59.435020+0000","last_undegraded":"2023-10-21T05:22:59.435020+0000","last_fullsized":"2023-10-21T05:22:59.435020+0000","mapping_epoch":12,"log_start":"41'200","ondisk_log_start":"41'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1009,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.540893+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:21:00.540893+0000","last_peered":"2023-10-21T05:21:00.540893+0000","last_clean":"2023-10-21T05:21:00.540893+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:21:00.540893+0000","last_undegraded":"2023-10-21T05:21:00.540893+0000","last_fullsized":"2023-10-21T05:21:00.540893+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'16995","reported_seq":23240,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:59.444688+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:22:59.444688+0000","last_peered":"2023-10-21T05:22:59.444688+0000","last_clean":"2023-10-21T05:22:59.444688+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:22:59.444688+0000","last_undegraded":"2023-10-21T05:22:59.444688+0000","last_fullsized":"2023-10-21T05:22:59.444688+0000","mapping_epoch":12,"log_start":"75'6900","ondisk_log_start":"75'6900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'3520","reported_seq":5408,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:59.321925+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:22:59.321925+0000","last_peered":"2023-10-21T05:22:59.321925+0000","last_clean":"2023-10-21T05:22:59.321925+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:22:59.321925+0000","last_undegraded":"2023-10-21T05:22:59.321925+0000","last_fullsized":"2023-10-21T05:22:59.321925+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3520,"log_dups_size":0,"ondisk_log_size":3520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'525191","reported_seq":1046641,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:00.475814+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:23:00.475814+0000","last_peered":"2023-10-21T05:23:00.475814+0000","last_clean":"2023-10-21T05:23:00.475814+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:23:00.475814+0000","last_undegraded":"2023-10-21T05:23:00.475814+0000","last_fullsized":"2023-10-21T05:23:00.475814+0000","mapping_epoch":15,"log_start":"88'515117","ondisk_log_start":"88'515117","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7129772032,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":522768,"num_read_kb":2060068,"num_write":521433,"num_write_kb":2085732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'17389","reported_seq":23864,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:59.809251+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:22:59.809251+0000","last_peered":"2023-10-21T05:22:59.809251+0000","last_clean":"2023-10-21T05:22:59.809251+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:22:59.809251+0000","last_undegraded":"2023-10-21T05:22:59.809251+0000","last_fullsized":"2023-10-21T05:22:59.809251+0000","mapping_epoch":12,"log_start":"75'7300","ondisk_log_start":"75'7300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'519111","reported_seq":1032854,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:00.009299+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:23:00.009299+0000","last_peered":"2023-10-21T05:23:00.009299+0000","last_clean":"2023-10-21T05:23:00.009299+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:23:00.009299+0000","last_undegraded":"2023-10-21T05:23:00.009299+0000","last_fullsized":"2023-10-21T05:23:00.009299+0000","mapping_epoch":15,"log_start":"88'509020","ondisk_log_start":"88'509020","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2502,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 353s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":7031480320,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":514922,"num_read_kb":2028532,"num_write":515807,"num_write_kb":2063228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6732,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:56.869202+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:22:56.869202+0000","last_peered":"2023-10-21T05:22:56.869202+0000","last_clean":"2023-10-21T05:22:56.869202+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:22:56.869202+0000","last_undegraded":"2023-10-21T05:22:56.869202+0000","last_fullsized":"2023-10-21T05:22:56.869202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'539341","reported_seq":1073527,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:00.468940+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:23:00.468940+0000","last_peered":"2023-10-21T05:23:00.468940+0000","last_clean":"2023-10-21T05:23:00.468940+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:23:00.468940+0000","last_undegraded":"2023-10-21T05:23:00.468940+0000","last_fullsized":"2023-10-21T05:23:00.468940+0000","mapping_epoch":15,"log_start":"88'529325","ondisk_log_start":"88'529325","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7330893824,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":535554,"num_read_kb":2110416,"num_write":535465,"num_write_kb":2141860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":168,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:29.952539+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:21:29.952539+0000","last_peered":"2023-10-21T05:21:29.952539+0000","last_clean":"2023-10-21T05:21:29.952539+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:21:29.952539+0000","last_undegraded":"2023-10-21T05:21:29.952539+0000","last_fullsized":"2023-10-21T05:21:29.952539+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'506564","reported_seq":1009185,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:59.983551+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:22:59.983551+0000","last_peered":"2023-10-21T05:22:59.983551+0000","last_clean":"2023-10-21T05:22:59.983551+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:22:59.983551+0000","last_undegraded":"2023-10-21T05:22:59.983551+0000","last_fullsized":"2023-10-21T05:22:59.983551+0000","mapping_epoch":15,"log_start":"88'496518","ondisk_log_start":"88'496518","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6873317395,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":503877,"num_read_kb":1985020,"num_write":502933,"num_write_kb":2011726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'3501","reported_seq":4925,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:59.331287+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:22:59.331287+0000","last_peered":"2023-10-21T05:22:59.331287+0000","last_clean":"2023-10-21T05:22:59.331287+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:22:59.331287+0000","last_undegraded":"2023-10-21T05:22:59.331287+0000","last_fullsized":"2023-10-21T05:22:59.331287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3501,"log_dups_size":0,"ondisk_log_size":3501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":265,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:24.583018+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:21:24.583018+0000","last_peered":"2023-10-21T05:21:24.583018+0000","last_clean":"2023-10-21T05:21:24.583018+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:21:24.583018+0000","last_undegraded":"2023-10-21T05:21:24.583018+0000","last_fullsized":"2023-10-21T05:21:24.583018+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28365463571,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2077121,"num_read_kb":8184036,"num_write":2075638,"num_write_kb":8302546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17003581440,"data_stored":19242250278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40227,"ondisk_log_size":40227,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37657,"ondisk_log_size":37657,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739309,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5807176,"kb_used_data":4228480,"kb_used_omap":375,"kb_used_meta":1578312,"kb_avail":88564664,"statfs":{"total":96636764160,"available":90690215936,"internally_reserved":0,"allocated":4329963520,"data_stored":4901813970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":384096,"internal_metadata":1616192416},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,9,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.25800000000000001,"5min":0.40699999999999997,"15min":0.38900000000000001},"min":{"1min":0.16500000000000001,"5min":0.161,"15min":0.14599999999999999},"max":{"1min":0.755,"5min":3.5329999999999999,"15min":4.9889999999999999},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.61799999999999999,"15min":0.51400000000000001},"min":{"1min":0.222,"5min":0.182,"15min":0.182},"max":{"1min":4.6429999999999998,"5min":6.992,"15min":6.992},"last":0.371}]},{"osd":2,"last update":"Sat Oct 21 05:22:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57199999999999995,"15min":0.504},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":5.3479999999999999,"5min":5.4390000000000001,"15min":5.4390000000000001},"last":0.316},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61599999999999999,"15min":0.54800000000000004},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":5.407,"5min":5.4740000000000002,"15min":5.4740000000000002},"last":0.34499999999999997}]}]},{"osd":2,"up_from":8,"seq":34359739308,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9498872,"kb_used_data":8302064,"kb_used_omap":1040,"kb_used_meta":1195759,"kb_avail":84872968,"statfs":{"total":96636764160,"available":86909919232,"internally_reserved":0,"allocated":8501313536,"data_stored":9620150458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1065100,"internal_metadata":1224458100},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.43099999999999999,"15min":0.47899999999999998},"min":{"1min":0.17599999999999999,"5min":0.16900000000000001,"15min":0.157},"max":{"1min":2.7610000000000001,"5min":7.234,"15min":7.234},"last":0.215},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42199999999999999,"15min":0.34999999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3380000000000001,"5min":4.4420000000000002,"15min":4.4420000000000002},"last":0.20200000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.38900000000000001,"15min":0.41499999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.335,"5min":4.335,"15min":4.335},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.42599999999999999,"15min":0.42799999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20000000000000001},"max":{"1min":4.2839999999999998,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.32800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739308,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5706804,"kb_used_data":4075948,"kb_used_omap":361,"kb_used_meta":1630486,"kb_avail":88665036,"statfs":{"total":96636764160,"available":90792996864,"internally_reserved":0,"allocated":4173770752,"data_stored":4719956538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":370285,"internal_metadata":1669618067},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,6,4,9,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.39900000000000002,"15min":0.41399999999999998},"min":{"1min":0.17699999999999999,"5min":0.161,"15min":0.161},"max":{"1min":3.919,"5min":3.919,"15min":5.0490000000000004},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.39500000000000002,"15min":0.45900000000000002},"min":{"1min":0.20399999999999999,"5min":0.19,"15min":0.17000000000000001},"max":{"1min":3.8980000000000001,"5min":3.8980000000000001,"15min":5.6989999999999998},"last":0.60199999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.42799999999999999,"15min":0.443},"min":{"1min":0.20899999999999999,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":3.9849999999999999,"5min":3.9849999999999999,"15min":6.4119999999999999},"last":0.443},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.45700000000000002},"min":{"1min":0.248,"5min":0.214,"15min":0.214},"max":{"1min":4.0220000000000002,"5min":5.8179999999999996,"15min":5.8179999999999996},"last":0.47199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4329054208,"data_stored":4901511168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4172754944,"data_stored":4719632403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8501772288,"data_stored":9621106707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:23:04.793 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:23:04.793 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:23:19.748 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:23:19.746+0000 7f874a7e7640 -1 reset not still connected to 0x5576704885b0 2023-10-21T05:23:19.767 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:23:19.766+0000 7f874a7e7640 -1 reset not still connected to 0x557670489110 2023-10-21T05:23:19.767 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:23:19.766+0000 7f874a7e7640 -1 reset not still connected to 0x5576713ac270 2023-10-21T05:23:19.768 INFO:tasks.ceph.osd.1.smithi119.stderr:2023-10-21T05:23:19.766+0000 7f874a7e7640 -1 reset not still connected to 0x5576713adba0 2023-10-21T05:23:24.795 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:23:25.340 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:23:25.342 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:23:25.384 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2440,"stamp":"2023-10-21T05:23:23.783396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28394974440,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2103365,"num_read_kb":8231689,"num_write":2084383,"num_write_kb":8336511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77815,"ondisk_log_size":77815,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21231432,"kb_used_data":16671088,"kb_used_omap":1776,"kb_used_meta":4558543,"kb_avail":261884088,"statfs":{"total":289910292480,"available":268169306112,"internally_reserved":0,"allocated":17071194112,"data_stored":19308067270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1819481,"internal_metadata":4667948199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,5,9,8,7,8,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14852096,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4116,"num_read_kb":16384,"num_write":4065,"num_write_kb":16260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005592"},"pg_stats":[{"pgid":"2.7","version":"89'10364","reported_seq":14230,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:19.754176+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:23:19.754176+0000","last_peered":"2023-10-21T05:23:19.754176+0000","last_clean":"2023-10-21T05:23:19.754176+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:23:19.754176+0000","last_undegraded":"2023-10-21T05:23:19.754176+0000","last_fullsized":"2023-10-21T05:23:19.754176+0000","mapping_epoch":12,"log_start":"41'300","ondisk_log_start":"41'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1010,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.559892+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:22:15.559892+0000","last_peered":"2023-10-21T05:22:15.559892+0000","last_clean":"2023-10-21T05:22:15.559892+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:22:15.559892+0000","last_undegraded":"2023-10-21T05:22:15.559892+0000","last_fullsized":"2023-10-21T05:22:15.559892+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'17101","reported_seq":23366,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:19.782654+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:23:19.782654+0000","last_peered":"2023-10-21T05:23:19.782654+0000","last_clean":"2023-10-21T05:23:19.782654+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:23:19.782654+0000","last_undegraded":"2023-10-21T05:23:19.782654+0000","last_fullsized":"2023-10-21T05:23:19.782654+0000","mapping_epoch":12,"log_start":"75'7100","ondisk_log_start":"75'7100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'3542","reported_seq":5434,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:19.729751+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:23:19.729751+0000","last_peered":"2023-10-21T05:23:19.729751+0000","last_clean":"2023-10-21T05:23:19.729751+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:23:19.729751+0000","last_undegraded":"2023-10-21T05:23:19.729751+0000","last_fullsized":"2023-10-21T05:23:19.729751+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'527197","reported_seq":1050680,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:20.536834+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:23:20.536834+0000","last_peered":"2023-10-21T05:23:20.536834+0000","last_clean":"2023-10-21T05:23:20.536834+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:23:20.536834+0000","last_undegraded":"2023-10-21T05:23:20.536834+0000","last_fullsized":"2023-10-21T05:23:20.536834+0000","mapping_epoch":15,"log_start":"88'517117","ondisk_log_start":"88'517117","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7137054720,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":524799,"num_read_kb":2068160,"num_write":523439,"num_write_kb":2093756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'17458","reported_seq":23953,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:19.293940+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:23:19.293940+0000","last_peered":"2023-10-21T05:23:19.293940+0000","last_clean":"2023-10-21T05:23:19.293940+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:23:19.293940+0000","last_undegraded":"2023-10-21T05:23:19.293940+0000","last_fullsized":"2023-10-21T05:23:19.293940+0000","mapping_epoch":12,"log_start":"75'7400","ondisk_log_start":"75'7400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'521122","reported_seq":1036906,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:20.090423+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:23:20.090423+0000","last_peered":"2023-10-21T05:23:20.090423+0000","last_clean":"2023-10-21T05:23:20.090423+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:23:20.090423+0000","last_undegraded":"2023-10-21T05:23:20.090423+0000","last_fullsized":"2023-10-21T05:23:20.090423+0000","mapping_epoch":15,"log_start":"88'511120","ondisk_log_start":"88'511120","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2677,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 374s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":7038312448,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":516947,"num_read_kb":2036584,"num_write":517708,"num_write_kb":2070832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6752,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:17.396984+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:23:17.396984+0000","last_peered":"2023-10-21T05:23:17.396984+0000","last_clean":"2023-10-21T05:23:17.396984+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:23:17.396984+0000","last_undegraded":"2023-10-21T05:23:17.396984+0000","last_fullsized":"2023-10-21T05:23:17.396984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'541410","reported_seq":1077668,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:20.530288+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:23:20.530288+0000","last_peered":"2023-10-21T05:23:20.530288+0000","last_clean":"2023-10-21T05:23:20.530288+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:23:20.530288+0000","last_undegraded":"2023-10-21T05:23:20.530288+0000","last_fullsized":"2023-10-21T05:23:20.530288+0000","mapping_epoch":15,"log_start":"88'531325","ondisk_log_start":"88'531325","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7338536960,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":537625,"num_read_kb":2118672,"num_write":537534,"num_write_kb":2150136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":168,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:29.952539+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:21:29.952539+0000","last_peered":"2023-10-21T05:21:29.952539+0000","last_clean":"2023-10-21T05:21:29.952539+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:21:29.952539+0000","last_undegraded":"2023-10-21T05:21:29.952539+0000","last_fullsized":"2023-10-21T05:21:29.952539+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'508527","reported_seq":1013100,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:20.088442+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:23:20.088442+0000","last_peered":"2023-10-21T05:23:20.088442+0000","last_clean":"2023-10-21T05:23:20.088442+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:23:20.088442+0000","last_undegraded":"2023-10-21T05:23:20.088442+0000","last_fullsized":"2023-10-21T05:23:20.088442+0000","mapping_epoch":15,"log_start":"88'498518","ondisk_log_start":"88'498518","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6880477203,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":505829,"num_read_kb":1992796,"num_write":504896,"num_write_kb":2019578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'3523","reported_seq":4951,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:19.738110+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:23:19.738110+0000","last_peered":"2023-10-21T05:23:19.738110+0000","last_clean":"2023-10-21T05:23:19.738110+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:23:19.738110+0000","last_undegraded":"2023-10-21T05:23:19.738110+0000","last_fullsized":"2023-10-21T05:23:19.738110+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3523,"log_dups_size":0,"ondisk_log_size":3523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":265,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:21:24.583018+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:21:24.583018+0000","last_peered":"2023-10-21T05:21:24.583018+0000","last_clean":"2023-10-21T05:21:24.583018+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:21:24.583018+0000","last_undegraded":"2023-10-21T05:21:24.583018+0000","last_fullsized":"2023-10-21T05:21:24.583018+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28394381331,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2085200,"num_read_kb":8216212,"num_write":2083577,"num_write_kb":8334302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17068621824,"data_stored":19307290662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40176,"ondisk_log_size":40176,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37540,"ondisk_log_size":37540,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739313,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5859436,"kb_used_data":4244644,"kb_used_omap":375,"kb_used_meta":1614408,"kb_avail":88512404,"statfs":{"total":96636764160,"available":90636701696,"internally_reserved":0,"allocated":4346515456,"data_stored":4918365906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":384096,"internal_metadata":1653154720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.38700000000000001,"15min":0.38900000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.14599999999999999},"max":{"1min":4.0519999999999996,"5min":4.0519999999999996,"15min":4.9889999999999999},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.57399999999999995,"15min":0.50600000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.42599999999999999,"5min":6.992,"15min":6.992},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.46400000000000002,"15min":0.505},"min":{"1min":0.254,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":0.36299999999999999,"5min":5.3479999999999999,"15min":5.4390000000000001},"last":0.31},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.50900000000000001,"15min":0.54900000000000004},"min":{"1min":0.28100000000000003,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.46800000000000003,"5min":5.407,"15min":5.4740000000000002},"last":0.36099999999999999}]}]},{"osd":2,"up_from":8,"seq":34359739312,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9613584,"kb_used_data":8334984,"kb_used_omap":1040,"kb_used_meta":1277551,"kb_avail":84758256,"statfs":{"total":96636764160,"available":86792454144,"internally_reserved":0,"allocated":8535023616,"data_stored":9653860538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1065100,"internal_metadata":1308213108},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:08 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45800000000000002,"15min":0.48899999999999999},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":2.9500000000000002,"5min":7.234,"15min":7.234},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45300000000000001,"15min":0.35299999999999998},"min":{"1min":0.20200000000000001,"5min":0.187,"15min":0.187},"max":{"1min":3.496,"5min":4.4420000000000002,"15min":4.4420000000000002},"last":0.29599999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:23:08 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.502,"15min":0.45500000000000002},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.45600000000000002,"15min":0.433},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20000000000000001},"max":{"1min":2.6800000000000002,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.32200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739312,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5758412,"kb_used_data":4091460,"kb_used_omap":361,"kb_used_meta":1666582,"kb_avail":88613428,"statfs":{"total":96636764160,"available":90740150272,"internally_reserved":0,"allocated":4189655040,"data_stored":4735840826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":370285,"internal_metadata":1706580371},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,3,4,6,8,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.39900000000000002,"15min":0.41399999999999998},"min":{"1min":0.17699999999999999,"5min":0.161,"15min":0.161},"max":{"1min":3.919,"5min":3.919,"15min":5.0490000000000004},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.39500000000000002,"15min":0.45900000000000002},"min":{"1min":0.20399999999999999,"5min":0.19,"15min":0.17000000000000001},"max":{"1min":3.8980000000000001,"5min":3.8980000000000001,"15min":5.6989999999999998},"last":0.32200000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.42799999999999999,"15min":0.443},"min":{"1min":0.20899999999999999,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":3.9849999999999999,"5min":3.9849999999999999,"15min":6.4119999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.45700000000000002},"min":{"1min":0.248,"5min":0.214,"15min":0.214},"max":{"1min":4.0220000000000002,"5min":5.8179999999999996,"15min":5.8179999999999996},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4345745408,"data_stored":4918202368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4188581888,"data_stored":4735459347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8534294528,"data_stored":9653628947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:23:25.386 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:23:25.387 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:23:45.388 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:23:45.889 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:23:45.890 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:23:45.923 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2451,"stamp":"2023-10-21T05:23:45.786905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28431514856,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2113283,"num_read_kb":8271161,"num_write":2094419,"num_write_kb":8376655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77870,"ondisk_log_size":77870,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21539520,"kb_used_data":16752104,"kb_used_omap":1776,"kb_used_meta":4785615,"kb_avail":261576000,"statfs":{"total":289910292480,"available":267853824000,"internally_reserved":0,"allocated":17154154496,"data_stored":19391027654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1819481,"internal_metadata":4900469927},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,8,8,10,10,13,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21594112,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5881,"num_read_kb":23404,"num_write":5944,"num_write_kb":23776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001953"},"pg_stats":[{"pgid":"2.7","version":"89'10400","reported_seq":14282,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:42.947603+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:23:42.947603+0000","last_peered":"2023-10-21T05:23:42.947603+0000","last_clean":"2023-10-21T05:23:42.947603+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:23:42.947603+0000","last_undegraded":"2023-10-21T05:23:42.947603+0000","last_fullsized":"2023-10-21T05:23:42.947603+0000","mapping_epoch":12,"log_start":"41'300","ondisk_log_start":"41'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1010,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.559892+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:22:15.559892+0000","last_peered":"2023-10-21T05:22:15.559892+0000","last_clean":"2023-10-21T05:22:15.559892+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:22:15.559892+0000","last_undegraded":"2023-10-21T05:22:15.559892+0000","last_fullsized":"2023-10-21T05:22:15.559892+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'17161","reported_seq":23451,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:42.948077+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:23:42.948077+0000","last_peered":"2023-10-21T05:23:42.948077+0000","last_clean":"2023-10-21T05:23:42.948077+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:23:42.948077+0000","last_undegraded":"2023-10-21T05:23:42.948077+0000","last_fullsized":"2023-10-21T05:23:42.948077+0000","mapping_epoch":12,"log_start":"75'7100","ondisk_log_start":"75'7100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'3554","reported_seq":5451,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:41.657978+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:23:41.657978+0000","last_peered":"2023-10-21T05:23:41.657978+0000","last_clean":"2023-10-21T05:23:41.657978+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:23:41.657978+0000","last_undegraded":"2023-10-21T05:23:41.657978+0000","last_fullsized":"2023-10-21T05:23:41.657978+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3554,"log_dups_size":0,"ondisk_log_size":3554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'529751","reported_seq":1055688,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:45.585706+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:23:45.585706+0000","last_peered":"2023-10-21T05:23:45.585706+0000","last_clean":"2023-10-21T05:23:45.585706+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:23:45.585706+0000","last_undegraded":"2023-10-21T05:23:45.585706+0000","last_fullsized":"2023-10-21T05:23:45.585706+0000","mapping_epoch":15,"log_start":"88'519717","ondisk_log_start":"88'519717","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7146205184,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":527253,"num_read_kb":2077920,"num_write":525993,"num_write_kb":2103972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'17538","reported_seq":24053,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:42.947741+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:23:42.947741+0000","last_peered":"2023-10-21T05:23:42.947741+0000","last_clean":"2023-10-21T05:23:42.947741+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:23:42.947741+0000","last_undegraded":"2023-10-21T05:23:42.947741+0000","last_fullsized":"2023-10-21T05:23:42.947741+0000","mapping_epoch":12,"log_start":"75'7500","ondisk_log_start":"75'7500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'523770","reported_seq":1042069,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:45.218134+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:23:45.218134+0000","last_peered":"2023-10-21T05:23:45.218134+0000","last_clean":"2023-10-21T05:23:45.218134+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:23:45.218134+0000","last_undegraded":"2023-10-21T05:23:45.218134+0000","last_fullsized":"2023-10-21T05:23:45.218134+0000","mapping_epoch":15,"log_start":"88'513721","ondisk_log_start":"88'513721","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2860,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 399s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":7047598080,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":519441,"num_read_kb":2046524,"num_write":520236,"num_write_kb":2080944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6777,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:42.948397+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:23:42.948397+0000","last_peered":"2023-10-21T05:23:42.948397+0000","last_clean":"2023-10-21T05:23:42.948397+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:23:42.948397+0000","last_undegraded":"2023-10-21T05:23:42.948397+0000","last_fullsized":"2023-10-21T05:23:42.948397+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'543926","reported_seq":1082759,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:45.595582+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:23:45.595582+0000","last_peered":"2023-10-21T05:23:45.595582+0000","last_clean":"2023-10-21T05:23:45.595582+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:23:45.595582+0000","last_undegraded":"2023-10-21T05:23:45.595582+0000","last_fullsized":"2023-10-21T05:23:45.595582+0000","mapping_epoch":15,"log_start":"88'533925","ondisk_log_start":"88'533925","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7347687424,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":540199,"num_read_kb":2128908,"num_write":540050,"num_write_kb":2160200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":169,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.577616+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:22:15.577616+0000","last_peered":"2023-10-21T05:22:15.577616+0000","last_clean":"2023-10-21T05:22:15.577616+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:22:15.577616+0000","last_undegraded":"2023-10-21T05:22:15.577616+0000","last_fullsized":"2023-10-21T05:22:15.577616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'510965","reported_seq":1017933,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:45.202340+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:23:45.202340+0000","last_peered":"2023-10-21T05:23:45.202340+0000","last_clean":"2023-10-21T05:23:45.202340+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:23:45.202340+0000","last_undegraded":"2023-10-21T05:23:45.202340+0000","last_fullsized":"2023-10-21T05:23:45.202340+0000","mapping_epoch":15,"log_start":"88'500918","ondisk_log_start":"88'500918","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6889431059,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":508225,"num_read_kb":2002332,"num_write":507334,"num_write_kb":2029330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'3535","reported_seq":4968,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:23:41.658771+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:23:41.658771+0000","last_peered":"2023-10-21T05:23:41.658771+0000","last_clean":"2023-10-21T05:23:41.658771+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:23:41.658771+0000","last_undegraded":"2023-10-21T05:23:41.658771+0000","last_fullsized":"2023-10-21T05:23:41.658771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3535,"log_dups_size":0,"ondisk_log_size":3535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":266,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.563738+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:22:15.563738+0000","last_peered":"2023-10-21T05:22:15.563738+0000","last_clean":"2023-10-21T05:22:15.563738+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:22:15.563738+0000","last_undegraded":"2023-10-21T05:22:15.563738+0000","last_fullsized":"2023-10-21T05:22:15.563738+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28430921747,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2095118,"num_read_kb":8255684,"num_write":2093613,"num_write_kb":8374446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17150971904,"data_stored":19389640742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40131,"ondisk_log_size":40131,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37640,"ondisk_log_size":37640,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739318,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5949288,"kb_used_data":4265632,"kb_used_omap":375,"kb_used_meta":1683272,"kb_avail":88422552,"statfs":{"total":96636764160,"available":90544693248,"internally_reserved":0,"allocated":4368007168,"data_stored":4939857618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":384096,"internal_metadata":1723671456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.38700000000000001,"15min":0.38900000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.14599999999999999},"max":{"1min":4.0519999999999996,"5min":4.0519999999999996,"15min":4.9889999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.57399999999999995,"15min":0.50600000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.42599999999999999,"5min":6.992,"15min":6.992},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Oct 21 05:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.46400000000000002,"15min":0.505},"min":{"1min":0.254,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":0.36299999999999999,"5min":5.3479999999999999,"15min":5.4390000000000001},"last":0.499},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.50900000000000001,"15min":0.54900000000000004},"min":{"1min":0.28100000000000003,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.46800000000000003,"5min":5.407,"15min":5.4740000000000002},"last":0.53000000000000003}]}]},{"osd":2,"up_from":8,"seq":34359739317,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9753772,"kb_used_data":8375268,"kb_used_omap":1040,"kb_used_meta":1377455,"kb_avail":84618068,"statfs":{"total":96636764160,"available":86648901632,"internally_reserved":0,"allocated":8576274432,"data_stored":9695111354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1065100,"internal_metadata":1410514804},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:08 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45800000000000002,"15min":0.48899999999999999},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":2.9500000000000002,"5min":7.234,"15min":7.234},"last":0.252},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45300000000000001,"15min":0.35299999999999998},"min":{"1min":0.20200000000000001,"5min":0.187,"15min":0.187},"max":{"1min":3.496,"5min":4.4420000000000002,"15min":4.4420000000000002},"last":0.28499999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:23:08 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.502,"15min":0.45500000000000002},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.45600000000000002,"15min":0.433},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20000000000000001},"max":{"1min":2.6800000000000002,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.33200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739317,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5836460,"kb_used_data":4111204,"kb_used_omap":361,"kb_used_meta":1724886,"kb_avail":88535380,"statfs":{"total":96636764160,"available":90660229120,"internally_reserved":0,"allocated":4209872896,"data_stored":4756058682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":370285,"internal_metadata":1766283667},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,2,4,8,13,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.39900000000000002,"15min":0.41399999999999998},"min":{"1min":0.17699999999999999,"5min":0.161,"15min":0.161},"max":{"1min":3.919,"5min":3.919,"15min":5.0490000000000004},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.39500000000000002,"15min":0.45900000000000002},"min":{"1min":0.20399999999999999,"5min":0.19,"15min":0.17000000000000001},"max":{"1min":3.8980000000000001,"5min":3.8980000000000001,"15min":5.6989999999999998},"last":0.245}]},{"osd":2,"last update":"Sat Oct 21 05:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.42799999999999999,"15min":0.443},"min":{"1min":0.20899999999999999,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":3.9849999999999999,"5min":3.9849999999999999,"15min":6.4119999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.45700000000000002},"min":{"1min":0.248,"5min":0.214,"15min":0.214},"max":{"1min":4.0220000000000002,"5min":5.8179999999999996,"15min":5.8179999999999996},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4366512128,"data_stored":4938969088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4208930816,"data_stored":4755808275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8575528960,"data_stored":9694863379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:23:45.925 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:23:45.926 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:24:05.927 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:24:06.411 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:24:06.412 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:24:06.436 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2461,"stamp":"2023-10-21T05:24:05.790291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28461038824,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2121643,"num_read_kb":8304421,"num_write":2102487,"num_write_kb":8408927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77889,"ondisk_log_size":77889,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21746768,"kb_used_data":16814964,"kb_used_omap":1776,"kb_used_meta":4929999,"kb_avail":261368752,"statfs":{"total":289910292480,"available":267641602048,"internally_reserved":0,"allocated":17218523136,"data_stored":19455396294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1819481,"internal_metadata":5048319143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,5,6,10,17,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22208512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6283,"num_read_kb":24988,"num_write":6051,"num_write_kb":24204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"2.7","version":"89'10457","reported_seq":14351,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:03.935211+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:24:03.935211+0000","last_peered":"2023-10-21T05:24:03.935211+0000","last_clean":"2023-10-21T05:24:03.935211+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:24:03.935211+0000","last_undegraded":"2023-10-21T05:24:03.935211+0000","last_fullsized":"2023-10-21T05:24:03.935211+0000","mapping_epoch":12,"log_start":"45'400","ondisk_log_start":"45'400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1010,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.559892+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:22:15.559892+0000","last_peered":"2023-10-21T05:22:15.559892+0000","last_clean":"2023-10-21T05:22:15.559892+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:22:15.559892+0000","last_undegraded":"2023-10-21T05:22:15.559892+0000","last_fullsized":"2023-10-21T05:22:15.559892+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'17256","reported_seq":23566,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:03.937017+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:24:03.937017+0000","last_peered":"2023-10-21T05:24:03.937017+0000","last_clean":"2023-10-21T05:24:03.937017+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:24:03.937017+0000","last_undegraded":"2023-10-21T05:24:03.937017+0000","last_fullsized":"2023-10-21T05:24:03.937017+0000","mapping_epoch":12,"log_start":"75'7200","ondisk_log_start":"75'7200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'3573","reported_seq":5474,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:03.575430+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:24:03.575430+0000","last_peered":"2023-10-21T05:24:03.575430+0000","last_clean":"2023-10-21T05:24:03.575430+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:24:03.575430+0000","last_undegraded":"2023-10-21T05:24:03.575430+0000","last_fullsized":"2023-10-21T05:24:03.575430+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3573,"log_dups_size":0,"ondisk_log_size":3573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'531760","reported_seq":1059746,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:05.671429+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:24:05.671429+0000","last_peered":"2023-10-21T05:24:05.671429+0000","last_clean":"2023-10-21T05:24:05.671429+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:24:05.671429+0000","last_undegraded":"2023-10-21T05:24:05.671429+0000","last_fullsized":"2023-10-21T05:24:05.671429+0000","mapping_epoch":15,"log_start":"89'521717","ondisk_log_start":"89'521717","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7153573888,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":529301,"num_read_kb":2086044,"num_write":528002,"num_write_kb":2112008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'17613","reported_seq":24148,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:05.626263+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:24:05.626263+0000","last_peered":"2023-10-21T05:24:05.626263+0000","last_clean":"2023-10-21T05:24:05.626263+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:24:05.626263+0000","last_undegraded":"2023-10-21T05:24:05.626263+0000","last_fullsized":"2023-10-21T05:24:05.626263+0000","mapping_epoch":12,"log_start":"75'7600","ondisk_log_start":"75'7600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'525852","reported_seq":1046272,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:05.313194+0000","last_change":"2023-10-21T05:17:06.050871+0000","last_active":"2023-10-21T05:24:05.313194+0000","last_peered":"2023-10-21T05:24:05.313194+0000","last_clean":"2023-10-21T05:24:05.313194+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:24:05.313194+0000","last_undegraded":"2023-10-21T05:24:05.313194+0000","last_fullsized":"2023-10-21T05:24:05.313194+0000","mapping_epoch":15,"log_start":"89'515821","ondisk_log_start":"89'515821","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:05:05.272834+0000","last_clean_scrub_stamp":"2023-10-21T04:05:05.272834+0000","objects_scrubbed":2997,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 419s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":7054815232,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":521550,"num_read_kb":2054916,"num_write":522232,"num_write_kb":2088928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'298","reported_seq":6797,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:03.428056+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:24:03.428056+0000","last_peered":"2023-10-21T05:24:03.428056+0000","last_clean":"2023-10-21T05:24:03.428056+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:24:03.428056+0000","last_undegraded":"2023-10-21T05:24:03.428056+0000","last_fullsized":"2023-10-21T05:24:03.428056+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'546038","reported_seq":1087018,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:05.665201+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:24:05.665201+0000","last_peered":"2023-10-21T05:24:05.665201+0000","last_clean":"2023-10-21T05:24:05.665201+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:24:05.665201+0000","last_undegraded":"2023-10-21T05:24:05.665201+0000","last_fullsized":"2023-10-21T05:24:05.665201+0000","mapping_epoch":15,"log_start":"89'536025","ondisk_log_start":"89'536025","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7355498496,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":542347,"num_read_kb":2137460,"num_write":542162,"num_write_kb":2168648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":169,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.577616+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:22:15.577616+0000","last_peered":"2023-10-21T05:22:15.577616+0000","last_clean":"2023-10-21T05:22:15.577616+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:22:15.577616+0000","last_undegraded":"2023-10-21T05:22:15.577616+0000","last_fullsized":"2023-10-21T05:22:15.577616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'512916","reported_seq":1021941,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:05.328835+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:24:05.328835+0000","last_peered":"2023-10-21T05:24:05.328835+0000","last_clean":"2023-10-21T05:24:05.328835+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:24:05.328835+0000","last_undegraded":"2023-10-21T05:24:05.328835+0000","last_fullsized":"2023-10-21T05:24:05.328835+0000","mapping_epoch":15,"log_start":"89'502818","ondisk_log_start":"89'502818","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6896558099,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":510280,"num_read_kb":2010524,"num_write":509285,"num_write_kb":2037134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'3554","reported_seq":4991,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:03.583006+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:24:03.583006+0000","last_peered":"2023-10-21T05:24:03.583006+0000","last_clean":"2023-10-21T05:24:03.583006+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:24:03.583006+0000","last_undegraded":"2023-10-21T05:24:03.583006+0000","last_fullsized":"2023-10-21T05:24:03.583006+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3554,"log_dups_size":0,"ondisk_log_size":3554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":266,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.563738+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:22:15.563738+0000","last_peered":"2023-10-21T05:22:15.563738+0000","last_clean":"2023-10-21T05:22:15.563738+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:22:15.563738+0000","last_undegraded":"2023-10-21T05:22:15.563738+0000","last_fullsized":"2023-10-21T05:22:15.563738+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28460445715,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2103478,"num_read_kb":8288944,"num_write":2101681,"num_write_kb":8406718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17217015808,"data_stored":19455684646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40185,"ondisk_log_size":40185,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37605,"ondisk_log_size":37605,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739322,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6001756,"kb_used_data":4282004,"kb_used_omap":375,"kb_used_meta":1719368,"kb_avail":88370084,"statfs":{"total":96636764160,"available":90490966016,"internally_reserved":0,"allocated":4384772096,"data_stored":4956622546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":384096,"internal_metadata":1760633760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,0,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.38700000000000001,"15min":0.38900000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.14599999999999999},"max":{"1min":4.0519999999999996,"5min":4.0519999999999996,"15min":4.9889999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.57399999999999995,"15min":0.50600000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.42599999999999999,"5min":6.992,"15min":6.992},"last":0.23400000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.46400000000000002,"15min":0.505},"min":{"1min":0.254,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":0.36299999999999999,"5min":5.3479999999999999,"15min":5.4390000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.50900000000000001,"15min":0.54900000000000004},"min":{"1min":0.28100000000000003,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.46800000000000003,"5min":5.407,"15min":5.4740000000000002},"last":0.32200000000000001}]}]},{"osd":2,"up_from":8,"seq":34359739321,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9870576,"kb_used_data":8405988,"kb_used_omap":1040,"kb_used_meta":1463535,"kb_avail":84501264,"statfs":{"total":96636764160,"available":86529294336,"internally_reserved":0,"allocated":8607731712,"data_stored":9726568634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1065100,"internal_metadata":1498660724},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:08 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45800000000000002,"15min":0.48899999999999999},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":2.9500000000000002,"5min":7.234,"15min":7.234},"last":0.246},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45300000000000001,"15min":0.35299999999999998},"min":{"1min":0.20200000000000001,"5min":0.187,"15min":0.187},"max":{"1min":3.496,"5min":4.4420000000000002,"15min":4.4420000000000002},"last":0.38600000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:23:08 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.502,"15min":0.45500000000000002},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.45600000000000002,"15min":0.433},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20000000000000001},"max":{"1min":2.6800000000000002,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.34200000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739321,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5874436,"kb_used_data":4126972,"kb_used_omap":361,"kb_used_meta":1747094,"kb_avail":88497404,"statfs":{"total":96636764160,"available":90621341696,"internally_reserved":0,"allocated":4226019328,"data_stored":4772205114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":370285,"internal_metadata":1789024659},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,6,4,10,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.45100000000000001,"15min":0.42499999999999999},"min":{"1min":0.16800000000000001,"5min":0.161,"15min":0.161},"max":{"1min":4.7830000000000004,"5min":4.7830000000000004,"15min":5.0490000000000004},"last":0.16600000000000001},{"interface":"front","average":{"1min":0.29399999999999998,"5min":0.39200000000000002,"15min":0.432},"min":{"1min":0.20100000000000001,"5min":0.19,"15min":0.17000000000000001},"max":{"1min":0.60199999999999998,"5min":3.8980000000000001,"15min":5.6989999999999998},"last":0.25}]},{"osd":2,"last update":"Sat Oct 21 05:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.70099999999999996,"5min":3.9849999999999999,"15min":6.4119999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.57799999999999996,"15min":0.45900000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.214},"max":{"1min":0.66000000000000003,"5min":5.8179999999999996,"15min":5.8179999999999996},"last":0.26800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4383399936,"data_stored":4955856896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4225101824,"data_stored":4771979283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8608514048,"data_stored":9727848467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:24:06.438 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:05:05.272834+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=5, tm_sec=5, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=10, tm_sec=54, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:24:06.439 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:24:06.984 INFO:teuthology.orchestra.run.smithi119.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:24:07.013 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:24:20.744 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:24:20.741+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dc680 2023-10-21T05:24:20.745 INFO:tasks.ceph.osd.0.smithi119.stderr:2023-10-21T05:24:20.741+0000 7fc9f3c29640 -1 reset not still connected to 0x55fa0b2dd110 2023-10-21T05:24:27.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-10-21T05:24:27.567 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2023-10-21T05:24:27.596 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2471,"stamp":"2023-10-21T05:24:25.796223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28492377320,"num_objects":12382,"num_object_clones":7168,"num_object_copies":24764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":2130438,"num_read_kb":8339429,"num_write":2111072,"num_write_kb":8443267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78028,"ondisk_log_size":78028,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22014620,"kb_used_data":16884356,"kb_used_omap":1776,"kb_used_meta":5128463,"kb_avail":261100900,"statfs":{"total":289910292480,"available":267367321600,"internally_reserved":0,"allocated":17289580544,"data_stored":19526453702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1819481,"internal_metadata":5251546279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,6,11,14,17,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24043520,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6774,"num_read_kb":26992,"num_write":6581,"num_write_kb":26324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001935"},"pg_stats":[{"pgid":"2.7","version":"89'10478","reported_seq":14384,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:23.911564+0000","last_change":"2023-10-21T05:11:06.831738+0000","last_active":"2023-10-21T05:24:23.911564+0000","last_peered":"2023-10-21T05:24:23.911564+0000","last_clean":"2023-10-21T05:24:23.911564+0000","last_became_active":"2023-10-21T04:05:01.187035+0000","last_became_peered":"2023-10-21T04:05:01.187035+0000","last_unstale":"2023-10-21T05:24:23.911564+0000","last_undegraded":"2023-10-21T05:24:23.911564+0000","last_fullsized":"2023-10-21T05:24:23.911564+0000","mapping_epoch":12,"log_start":"45'400","ondisk_log_start":"45'400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'9015","last_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_deep_scrub":"79'9015","last_deep_scrub_stamp":"2023-10-21T05:11:06.831466+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.831466+0000","objects_scrubbed":15,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:05:53.753491+0000","scrub_duration":0.127982182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2466,"num_read_kb":2058,"num_write":84,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"79'40","reported_seq":1010,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.559892+0000","last_change":"2023-10-21T05:11:08.750236+0000","last_active":"2023-10-21T05:22:15.559892+0000","last_peered":"2023-10-21T05:22:15.559892+0000","last_clean":"2023-10-21T05:22:15.559892+0000","last_became_active":"2023-10-21T04:05:01.187081+0000","last_became_peered":"2023-10-21T04:05:01.187081+0000","last_unstale":"2023-10-21T05:22:15.559892+0000","last_undegraded":"2023-10-21T05:22:15.559892+0000","last_fullsized":"2023-10-21T05:22:15.559892+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'40","last_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_deep_scrub":"79'40","last_deep_scrub_stamp":"2023-10-21T05:11:08.750013+0000","last_clean_scrub_stamp":"2023-10-21T05:11:08.750013+0000","objects_scrubbed":9,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:25:32.270241+0000","scrub_duration":0.022578358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1457,"num_read_kb":1434,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1412,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'17291","reported_seq":23621,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:23.912229+0000","last_change":"2023-10-21T05:11:03.927135+0000","last_active":"2023-10-21T05:24:23.912229+0000","last_peered":"2023-10-21T05:24:23.912229+0000","last_clean":"2023-10-21T05:24:23.912229+0000","last_became_active":"2023-10-21T04:05:01.187215+0000","last_became_peered":"2023-10-21T04:05:01.187215+0000","last_unstale":"2023-10-21T05:24:23.912229+0000","last_undegraded":"2023-10-21T05:24:23.912229+0000","last_fullsized":"2023-10-21T05:24:23.912229+0000","mapping_epoch":12,"log_start":"75'7200","ondisk_log_start":"75'7200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14879","last_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_deep_scrub":"79'14879","last_deep_scrub_stamp":"2023-10-21T05:11:03.926849+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.926849+0000","objects_scrubbed":15,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:26:09.425222+0000","scrub_duration":0.12612614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3775,"num_read_kb":3176,"num_write":119,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2807,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'3580","reported_seq":5485,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:22.082016+0000","last_change":"2023-10-21T05:11:05.767912+0000","last_active":"2023-10-21T05:24:22.082016+0000","last_peered":"2023-10-21T05:24:22.082016+0000","last_clean":"2023-10-21T05:24:22.082016+0000","last_became_active":"2023-10-21T04:05:01.188247+0000","last_became_peered":"2023-10-21T04:05:01.188247+0000","last_unstale":"2023-10-21T05:24:22.082016+0000","last_undegraded":"2023-10-21T05:24:22.082016+0000","last_fullsized":"2023-10-21T05:24:22.082016+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3084","last_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_deep_scrub":"79'3084","last_deep_scrub_stamp":"2023-10-21T05:11:05.767744+0000","last_clean_scrub_stamp":"2023-10-21T05:11:05.767744+0000","objects_scrubbed":11,"log_size":3580,"log_dups_size":0,"ondisk_log_size":3580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:06:40.827052+0000","scrub_duration":0.021799267000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1290,"num_read_kb":1117,"num_write":65,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1049,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'533958","reported_seq":1064185,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:25.762662+0000","last_change":"2023-10-21T05:17:04.634511+0000","last_active":"2023-10-21T05:24:25.762662+0000","last_peered":"2023-10-21T05:24:25.762662+0000","last_clean":"2023-10-21T05:24:25.762662+0000","last_became_active":"2023-10-21T04:05:06.308948+0000","last_became_peered":"2023-10-21T04:05:06.308948+0000","last_unstale":"2023-10-21T05:24:25.762662+0000","last_undegraded":"2023-10-21T05:24:25.762662+0000","last_fullsized":"2023-10-21T05:24:25.762662+0000","mapping_epoch":15,"log_start":"89'523917","ondisk_log_start":"89'523917","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'489506","last_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_deep_scrub":"84'489506","last_deep_scrub_stamp":"2023-10-21T05:17:04.634336+0000","last_clean_scrub_stamp":"2023-10-21T05:17:04.634336+0000","objects_scrubbed":3090,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":359,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:03:05.290147+0000","scrub_duration":358.10206996599999,"objects_trimmed":0,"snaptrim_duration":0.015688061999999999,"stat_sum":{"num_bytes":7161618432,"num_objects":3090,"num_object_clones":1803,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":531543,"num_read_kb":2094952,"num_write":530200,"num_write_kb":2120800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'17673","reported_seq":24228,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:24.961329+0000","last_change":"2023-10-21T05:11:03.242493+0000","last_active":"2023-10-21T05:24:24.961329+0000","last_peered":"2023-10-21T05:24:24.961329+0000","last_clean":"2023-10-21T05:24:24.961329+0000","last_became_active":"2023-10-21T04:05:01.186822+0000","last_became_peered":"2023-10-21T04:05:01.186822+0000","last_unstale":"2023-10-21T05:24:24.961329+0000","last_undegraded":"2023-10-21T05:24:24.961329+0000","last_fullsized":"2023-10-21T05:24:24.961329+0000","mapping_epoch":12,"log_start":"75'7600","ondisk_log_start":"75'7600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'15035","last_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_deep_scrub":"79'15035","last_deep_scrub_stamp":"2023-10-21T05:11:03.242272+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.242272+0000","objects_scrubbed":14,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:24:07.624677+0000","scrub_duration":0.67187215099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4096,"num_read_kb":3451,"num_write":164,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4384,"num_omap_keys":77,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'528060","reported_seq":1050613,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:25.388439+0000","last_change":"2023-10-21T05:24:09.337331+0000","last_active":"2023-10-21T05:24:25.388439+0000","last_peered":"2023-10-21T05:24:25.388439+0000","last_clean":"2023-10-21T05:24:25.388439+0000","last_became_active":"2023-10-21T04:05:06.308925+0000","last_became_peered":"2023-10-21T04:05:06.308925+0000","last_unstale":"2023-10-21T05:24:25.388439+0000","last_undegraded":"2023-10-21T05:24:25.388439+0000","last_fullsized":"2023-10-21T05:24:25.388439+0000","mapping_epoch":15,"log_start":"89'518021","ondisk_log_start":"89'518021","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'526261","last_scrub_stamp":"2023-10-21T05:24:09.336948+0000","last_deep_scrub":"89'526261","last_deep_scrub_stamp":"2023-10-21T05:24:09.336948+0000","last_clean_scrub_stamp":"2023-10-21T05:24:09.336948+0000","objects_scrubbed":3026,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":424,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:26:40.061657+0000","scrub_duration":423.285916547,"objects_trimmed":0,"snaptrim_duration":0.113845933,"stat_sum":{"num_bytes":7062736896,"num_objects":3026,"num_object_clones":1755,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":523676,"num_read_kb":2063388,"num_write":524421,"num_write_kb":2097684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"89'303","reported_seq":6818,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:23.973928+0000","last_change":"2023-10-21T05:11:06.603268+0000","last_active":"2023-10-21T05:24:23.973928+0000","last_peered":"2023-10-21T05:24:23.973928+0000","last_clean":"2023-10-21T05:24:23.973928+0000","last_became_active":"2023-10-21T04:05:01.189593+0000","last_became_peered":"2023-10-21T04:05:01.189593+0000","last_unstale":"2023-10-21T05:24:23.973928+0000","last_undegraded":"2023-10-21T05:24:23.973928+0000","last_fullsized":"2023-10-21T05:24:23.973928+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'293","last_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_deep_scrub":"79'293","last_deep_scrub_stamp":"2023-10-21T05:11:06.603086+0000","last_clean_scrub_stamp":"2023-10-21T05:11:06.603086+0000","objects_scrubbed":11,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T15:04:25.593657+0000","scrub_duration":0.017603456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4036,"num_read_kb":3399,"num_write":142,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3024,"num_omap_keys":57,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'548219","reported_seq":1091500,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:25.740573+0000","last_change":"2023-10-21T05:17:43.907329+0000","last_active":"2023-10-21T05:24:25.740573+0000","last_peered":"2023-10-21T05:24:25.740573+0000","last_clean":"2023-10-21T05:24:25.740573+0000","last_became_active":"2023-10-21T04:05:06.311724+0000","last_became_peered":"2023-10-21T04:05:06.311724+0000","last_unstale":"2023-10-21T05:24:25.740573+0000","last_undegraded":"2023-10-21T05:24:25.740573+0000","last_fullsized":"2023-10-21T05:24:25.740573+0000","mapping_epoch":15,"log_start":"89'538125","ondisk_log_start":"89'538125","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'506802","last_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_deep_scrub":"85'506802","last_deep_scrub_stamp":"2023-10-21T05:17:43.907132+0000","last_clean_scrub_stamp":"2023-10-21T05:17:43.907132+0000","objects_scrubbed":3185,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":399,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:34:14.807919+0000","scrub_duration":398.36530872700001,"objects_trimmed":0,"snaptrim_duration":0.0089595439999999998,"stat_sum":{"num_bytes":7363391488,"num_objects":3185,"num_object_clones":1864,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":544646,"num_read_kb":2146624,"num_write":544343,"num_write_kb":2177372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"79'14","reported_seq":169,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.577616+0000","last_change":"2023-10-21T05:11:03.630166+0000","last_active":"2023-10-21T05:22:15.577616+0000","last_peered":"2023-10-21T05:22:15.577616+0000","last_clean":"2023-10-21T05:22:15.577616+0000","last_became_active":"2023-10-21T04:05:01.189512+0000","last_became_peered":"2023-10-21T04:05:01.189512+0000","last_unstale":"2023-10-21T05:22:15.577616+0000","last_undegraded":"2023-10-21T05:22:15.577616+0000","last_fullsized":"2023-10-21T05:22:15.577616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'14","last_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_deep_scrub":"79'14","last_deep_scrub_stamp":"2023-10-21T05:11:03.629905+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.629905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:14:27.949246+0000","scrub_duration":0.089676989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'514933","reported_seq":1026086,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:25.413119+0000","last_change":"2023-10-21T05:17:02.295237+0000","last_active":"2023-10-21T05:24:25.413119+0000","last_peered":"2023-10-21T05:24:25.413119+0000","last_clean":"2023-10-21T05:24:25.413119+0000","last_became_active":"2023-10-21T04:05:06.308755+0000","last_became_peered":"2023-10-21T04:05:06.308755+0000","last_unstale":"2023-10-21T05:24:25.413119+0000","last_undegraded":"2023-10-21T05:24:25.413119+0000","last_fullsized":"2023-10-21T05:24:25.413119+0000","mapping_epoch":15,"log_start":"89'504918","ondisk_log_start":"89'504918","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'472049","last_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_deep_scrub":"84'472049","last_deep_scrub_stamp":"2023-10-21T05:17:02.295051+0000","last_clean_scrub_stamp":"2023-10-21T05:17:02.295051+0000","objects_scrubbed":2988,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":353,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:32:56.844504+0000","scrub_duration":352.54869644399997,"objects_trimmed":0,"snaptrim_duration":0.0073793979999999997,"stat_sum":{"num_bytes":6904037395,"num_objects":2988,"num_object_clones":1746,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":512408,"num_read_kb":2018988,"num_write":511302,"num_write_kb":2045202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'3561","reported_seq":5002,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:24:22.090056+0000","last_change":"2023-10-21T05:11:03.284152+0000","last_active":"2023-10-21T05:24:22.090056+0000","last_peered":"2023-10-21T05:24:22.090056+0000","last_clean":"2023-10-21T05:24:22.090056+0000","last_became_active":"2023-10-21T04:05:01.185939+0000","last_became_peered":"2023-10-21T04:05:01.185939+0000","last_unstale":"2023-10-21T05:24:22.090056+0000","last_undegraded":"2023-10-21T05:24:22.090056+0000","last_fullsized":"2023-10-21T05:24:22.090056+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'3061","last_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_deep_scrub":"79'3061","last_deep_scrub_stamp":"2023-10-21T05:11:03.283950+0000","last_clean_scrub_stamp":"2023-10-21T05:11:03.283950+0000","objects_scrubbed":9,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:07.196499+0000","scrub_duration":0.51436914899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":900,"num_read_kb":739,"num_write":37,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"79'99","reported_seq":266,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:22:15.563738+0000","last_change":"2023-10-21T05:11:01.853225+0000","last_active":"2023-10-21T05:22:15.563738+0000","last_peered":"2023-10-21T05:22:15.563738+0000","last_clean":"2023-10-21T05:22:15.563738+0000","last_became_active":"2023-10-21T04:04:56.454691+0000","last_became_peered":"2023-10-21T04:04:56.454691+0000","last_unstale":"2023-10-21T05:22:15.563738+0000","last_undegraded":"2023-10-21T05:22:15.563738+0000","last_fullsized":"2023-10-21T05:22:15.563738+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'99","last_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_deep_scrub":"79'99","last_deep_scrub_stamp":"2023-10-21T05:11:01.852844+0000","last_clean_scrub_stamp":"2023-10-21T05:11:01.852844+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:40:01.469683+0000","scrub_duration":0.089296405999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28491784211,"num_objects":12289,"num_object_clones":7168,"num_object_copies":24578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12289,"num_whiteouts":0,"num_read":2112273,"num_read_kb":8323952,"num_write":2110266,"num_write_kb":8441058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17287368704,"data_stored":19526037542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40189,"ondisk_log_size":40189,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2741,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":18059,"num_read_kb":15386,"num_write":669,"num_write_kb":315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15399,"num_omap_keys":331,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":5482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35982,"internal_metadata":0},"log_size":37740,"ondisk_log_size":37740,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359739326,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6073312,"kb_used_data":4299416,"kb_used_omap":375,"kb_used_meta":1773512,"kb_avail":88298528,"statfs":{"total":96636764160,"available":90417692672,"internally_reserved":0,"allocated":4402601984,"data_stored":4974452434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":384096,"internal_metadata":1816077216},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,2,7,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.42599999999999999,"15min":0.41099999999999998},"min":{"1min":0.189,"5min":0.161,"15min":0.161},"max":{"1min":5.2779999999999996,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.193},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.49199999999999999,"15min":0.51000000000000001},"min":{"1min":0.20200000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.036,"5min":5.923,"15min":6.992},"last":0.29699999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.48399999999999999,"15min":0.51800000000000002},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":4.7930000000000001,"5min":5.3479999999999999,"15min":5.4390000000000001},"last":0.317},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.52900000000000003,"15min":0.56200000000000006},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25},"max":{"1min":4.8259999999999996,"5min":5.407,"15min":5.4740000000000002},"last":0.34899999999999998}]}]},{"osd":2,"up_from":8,"seq":34359739325,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10012708,"kb_used_data":8441308,"kb_used_omap":1040,"kb_used_meta":1570351,"kb_avail":84359132,"statfs":{"total":96636764160,"available":86383751168,"internally_reserved":0,"allocated":8643899392,"data_stored":9762736314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1065100,"internal_metadata":1608040308},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:09 2023","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.371,"15min":0.436},"min":{"1min":0.17499999999999999,"5min":0.156,"15min":0.156},"max":{"1min":0.45400000000000001,"5min":2.9500000000000002,"15min":7.234},"last":2.5030000000000001},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.42599999999999999,"15min":0.35499999999999998},"min":{"1min":0.20300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.55500000000000005,"5min":4.4420000000000002,"15min":4.4420000000000002},"last":0.52100000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:24:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.51000000000000001,"15min":0.45700000000000002},"min":{"1min":0.23999999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.49399999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.45000000000000001,"15min":0.42699999999999999},"min":{"1min":0.23000000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.51300000000000001,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.44600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739325,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5928600,"kb_used_data":4143632,"kb_used_omap":361,"kb_used_meta":1784598,"kb_avail":88443240,"statfs":{"total":96636764160,"available":90565877760,"internally_reserved":0,"allocated":4243079168,"data_stored":4789264954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":370285,"internal_metadata":1827428755},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,6,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.45100000000000001,"15min":0.42499999999999999},"min":{"1min":0.16800000000000001,"5min":0.161,"15min":0.161},"max":{"1min":4.7830000000000004,"5min":4.7830000000000004,"15min":5.0490000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.29399999999999998,"5min":0.39200000000000002,"15min":0.432},"min":{"1min":0.20100000000000001,"5min":0.19,"15min":0.17000000000000001},"max":{"1min":0.60199999999999998,"5min":3.8980000000000001,"15min":5.6989999999999998},"last":0.22500000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.23300000000000001,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.70099999999999996,"5min":3.9849999999999999,"15min":6.4119999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.57799999999999996,"15min":0.45900000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.214},"max":{"1min":0.66000000000000003,"5min":5.8179999999999996,"15min":5.8179999999999996},"last":0.251}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":1888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14100,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":2741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16907,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4975,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4401348608,"data_stored":4973805568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4242337792,"data_stored":4789215251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8643682304,"data_stored":9763016723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:24:27.599 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-21T05:24:28.205 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-21T05:24:28.205 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-21T05:24:28.206 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-21T05:24:28.206 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:24:32.878 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-21T05:24:34.309 INFO:tasks.ceph.osd.0:Stopped 2023-10-21T05:24:34.310 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-21T05:24:34.310 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:24:38.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-21T05:24:38.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-21T05:24:40.412 INFO:tasks.ceph.osd.1:Stopped 2023-10-21T05:24:40.413 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-21T05:24:40.413 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:24:43.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-21T05:24:43.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-21T05:24:46.516 INFO:tasks.ceph.osd.2:Stopped 2023-10-21T05:24:46.516 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-21T05:24:46.517 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-21T05:24:46.517 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:24:46.548 INFO:tasks.ceph.mgr.x:Stopped 2023-10-21T05:24:46.548 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-21T05:24:46.549 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-21T05:24:46.549 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:24:46.611 INFO:tasks.ceph.mon.a:Stopped 2023-10-21T05:24:46.612 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-21T05:24:46.613 DEBUG:teuthology.orchestra.run.smithi119:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-21T05:24:46.733 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi119.front.sepia.ceph.com 2023-10-21T05:24:46.733 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-21T05:24:48.386 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi119.front.sepia.ceph.com 2023-10-21T05:24:48.386 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-21T05:24:48.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-21T05:24:48.756 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi119.front.sepia.ceph.com 2023-10-21T05:24:48.757 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-21T05:24:49.265 INFO:tasks.ceph:Archiving mon data... 2023-10-21T05:24:49.267 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433375/data/mon.a.tgz 2023-10-21T05:24:49.267 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2023-10-21T05:24:49.273 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.Cpw9U9l7oi 2023-10-21T05:24:49.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Cpw9U9l7oi 2023-10-21T05:24:49.419 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.Cpw9U9l7oi 2023-10-21T05:24:49.495 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.Cpw9U9l7oi is 421KB 2023-10-21T05:24:49.553 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.Cpw9U9l7oi 2023-10-21T05:24:49.558 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-21T05:24:49.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-21T05:24:49.634 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-21T05:24:49.634 DEBUG:teuthology.orchestra.run.smithi119:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-21T05:24:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-21T05:24:49.726 INFO:tasks.ceph:Archiving crash dumps... 2023-10-21T05:24:49.728 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433375/remote/smithi119/crash 2023-10-21T05:24:49.729 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-21T05:24:49.739 INFO:tasks.ceph:Compressing logs... 2023-10-21T05:24:49.739 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-21T05:24:49.800 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40020.log 2023-10-21T05:24:49.800 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45330.log 2023-10-21T05:24:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45876.log 2023-10-21T05:24:49.963 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40020.log.gz 2023-10-21T05:24:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54439.log 2023-10-21T05:24:49.964 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45330.log: /var/log/ceph/ceph-client.admin.45876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47298.log 2023-10-21T05:24:49.964 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45876.log.gz 2023-10-21T05:24:49.964 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45330.log.gz 2023-10-21T05:24:49.965 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52176.log 2023-10-21T05:24:49.965 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54439.log.gz 2023-10-21T05:24:49.965 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46178.log 2023-10-21T05:24:49.965 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47298.log.gz 2023-10-21T05:24:49.966 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34760.log 2023-10-21T05:24:49.966 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52176.log.gz 2023-10-21T05:24:49.966 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53587.log 2023-10-21T05:24:49.967 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46178.log.gz 2023-10-21T05:24:49.967 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.34760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54172.log 2023-10-21T05:24:49.967 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34760.log.gz 2023-10-21T05:24:49.967 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52838.log 2023-10-21T05:24:49.968 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53587.log.gz 2023-10-21T05:24:49.968 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44754.log 2023-10-21T05:24:49.968 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54172.log.gz 2023-10-21T05:24:49.968 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48556.log 2023-10-21T05:24:49.969 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52838.log.gz 2023-10-21T05:24:49.969 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52266.log 2023-10-21T05:24:49.969 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44754.log.gz 2023-10-21T05:24:49.970 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50742.log 2023-10-21T05:24:49.970 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48556.log.gz 2023-10-21T05:24:49.970 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41673.log 2023-10-21T05:24:49.970 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52266.log.gz 2023-10-21T05:24:49.971 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42918.log 2023-10-21T05:24:49.971 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39405.log 2023-10-21T05:24:49.971 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54281.log 2023-10-21T05:24:49.972 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42918.log.gz 2023-10-21T05:24:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53248.log 2023-10-21T05:24:49.972 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39405.log.gz 2023-10-21T05:24:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43943.log 2023-10-21T05:24:49.973 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54281.log.gz 2023-10-21T05:24:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51686.log 2023-10-21T05:24:49.973 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53248.log.gz 2023-10-21T05:24:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51381.log 2023-10-21T05:24:49.974 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43943.log.gz 2023-10-21T05:24:49.974 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50620.log 2023-10-21T05:24:49.974 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51686.log.gz 2023-10-21T05:24:49.975 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47760.log 2023-10-21T05:24:49.975 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51381.log.gz 2023-10-21T05:24:49.975 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54655.log 2023-10-21T05:24:49.975 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50620.log.gz 2023-10-21T05:24:49.976 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46630.log 2023-10-21T05:24:49.976 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47760.log.gz 2023-10-21T05:24:49.976 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46691.log 2023-10-21T05:24:49.976 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54655.log.gz 2023-10-21T05:24:49.977 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53274.log 2023-10-21T05:24:49.977 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46630.log.gz 2023-10-21T05:24:49.977 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45387.log 2023-10-21T05:24:49.978 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46691.log.gz 2023-10-21T05:24:49.978 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53151.log 2023-10-21T05:24:49.978 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53274.log.gz 2023-10-21T05:24:49.978 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47513.log 2023-10-21T05:24:49.979 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45387.log.gz 2023-10-21T05:24:49.979 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44315.log 2023-10-21T05:24:49.979 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53151.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53151.log.gz 2023-10-21T05:24:49.979 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39145.log 2023-10-21T05:24:49.980 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47513.log.gz 2023-10-21T05:24:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44315.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.44315.log.gz 2023-10-21T05:24:49.980 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/ceph-client.admin.42888.log 2023-10-21T05:24:49.981 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44291.log 2023-10-21T05:24:49.981 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39145.log.gz 2023-10-21T05:24:49.981 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46058.log 2023-10-21T05:24:49.981 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42888.log.gz 2023-10-21T05:24:49.982 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50958.log 2023-10-21T05:24:49.982 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44291.log.gz 2023-10-21T05:24:49.982 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40742.log 2023-10-21T05:24:49.982 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46058.log.gz 2023-10-21T05:24:49.983 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40273.log 2023-10-21T05:24:49.983 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50958.log.gz 2023-10-21T05:24:49.983 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52966.log 2023-10-21T05:24:49.984 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40742.log.gz 2023-10-21T05:24:49.984 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53694.log 2023-10-21T05:24:49.984 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40273.log.gz 2023-10-21T05:24:49.984 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42704.log 2023-10-21T05:24:49.985 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52966.log.gz 2023-10-21T05:24:49.985 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50046.log 2023-10-21T05:24:49.985 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53694.log.gz 2023-10-21T05:24:49.985 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45002.log 2023-10-21T05:24:49.986 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42704.log.gz 2023-10-21T05:24:49.986 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49530.log 2023-10-21T05:24:49.986 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50046.log.gz 2023-10-21T05:24:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40256.log 2023-10-21T05:24:49.987 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45002.log.gz 2023-10-21T05:24:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51471.log 2023-10-21T05:24:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49530.log.gz 2023-10-21T05:24:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42118.log 2023-10-21T05:24:49.988 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40256.log.gz 2023-10-21T05:24:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50229.log 2023-10-21T05:24:49.989 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51471.log.gz 2023-10-21T05:24:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:gzip/var/log/ceph/ceph-client.admin.42118.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52911.log 2023-10-21T05:24:49.989 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42118.log.gz 2023-10-21T05:24:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40310.log 2023-10-21T05:24:49.990 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50229.log.gz 2023-10-21T05:24:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50260.log 2023-10-21T05:24:49.990 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52911.log.gz 2023-10-21T05:24:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48494.log 2023-10-21T05:24:49.991 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40310.log.gz 2023-10-21T05:24:49.991 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45030.log 2023-10-21T05:24:49.991 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50260.log.gz 2023-10-21T05:24:49.991 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48494.log.gz 2023-10-21T05:24:49.992 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-21T05:24:49.992 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47117.log 2023-10-21T05:24:49.992 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45030.log.gz 2023-10-21T05:24:49.993 INFO:teuthology.orchestra.run.smithi119.stderr: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.50742.log.gz 2023-10-21T05:24:49.993 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41121.log 2023-10-21T05:24:49.993 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45478.log 2023-10-21T05:24:49.993 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47117.log.gz 2023-10-21T05:24:49.994 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44866.log 2023-10-21T05:24:49.994 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41121.log: /var/log/ceph/ceph-client.admin.45478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43882.log 2023-10-21T05:24:49.994 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45478.log.gz 2023-10-21T05:24:49.995 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52296.log 2023-10-21T05:24:49.995 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44866.log.gz 2023-10-21T05:24:49.995 INFO:teuthology.orchestra.run.smithi119.stderr:gzip/var/log/ceph/ceph-client.admin.43882.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49590.log 2023-10-21T05:24:49.995 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43882.log.gz 2023-10-21T05:24:49.996 INFO:teuthology.orchestra.run.smithi119.stderr:gzip/var/log/ceph/ceph-client.admin.52296.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49195.log 2023-10-21T05:24:49.996 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52296.log.gz 2023-10-21T05:24:49.996 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48000.log 2023-10-21T05:24:49.996 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49590.log.gz 2023-10-21T05:24:49.997 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54146.log 2023-10-21T05:24:49.997 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49195.log.gz 2023-10-21T05:24:49.997 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38932.log 2023-10-21T05:24:49.997 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48000.log.gz 2023-10-21T05:24:49.998 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54146.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54146.log.gz -5 2023-10-21T05:24:49.998 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49772.log 2023-10-21T05:24:49.998 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.38932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44263.log 2023-10-21T05:24:49.999 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38932.log.gz 2023-10-21T05:24:49.999 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52389.log 2023-10-21T05:24:49.999 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49772.log.gz 2023-10-21T05:24:49.999 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40455.log 2023-10-21T05:24:50.000 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44263.log.gz 2023-10-21T05:24:50.000 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54523.log 2023-10-21T05:24:50.000 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52389.log.gz 2023-10-21T05:24:50.000 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40131.log 2023-10-21T05:24:50.001 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40455.log.gz 2023-10-21T05:24:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43346.log 2023-10-21T05:24:50.001 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54523.log.gz 2023-10-21T05:24:50.002 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40725.log 2023-10-21T05:24:50.002 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40131.log.gz 2023-10-21T05:24:50.002 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43485.log 2023-10-21T05:24:50.002 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43346.log.gz 2023-10-21T05:24:50.003 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53456.log 2023-10-21T05:24:50.003 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40725.log.gz 2023-10-21T05:24:50.003 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43564.log 2023-10-21T05:24:50.003 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43485.log.gz 2023-10-21T05:24:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47700.log 2023-10-21T05:24:50.004 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53456.log.gz 2023-10-21T05:24:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54550.log 2023-10-21T05:24:50.005 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43564.log.gz 2023-10-21T05:24:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54761.log 2023-10-21T05:24:50.005 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47700.log.gz 2023-10-21T05:24:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53038.log 2023-10-21T05:24:50.006 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54550.log.gz 2023-10-21T05:24:50.006 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53207.log 2023-10-21T05:24:50.006 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54761.log.gz 2023-10-21T05:24:50.006 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47483.log 2023-10-21T05:24:50.007 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53038.log.gz 2023-10-21T05:24:50.007 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43411.log 2023-10-21T05:24:50.007 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53207.log.gz 2023-10-21T05:24:50.007 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50016.log 2023-10-21T05:24:50.008 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47483.log.gz 2023-10-21T05:24:50.008 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42255.log 2023-10-21T05:24:50.008 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43411.log.gz 2023-10-21T05:24:50.009 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42500.log 2023-10-21T05:24:50.009 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50016.log.gz 2023-10-21T05:24:50.009 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52801.log 2023-10-21T05:24:50.009 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42255.log.gz 2023-10-21T05:24:50.010 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41673.log.gz 2023-10-21T05:24:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50470.log 2023-10-21T05:24:50.010 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42500.log.gz 2023-10-21T05:24:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41841.log 2023-10-21T05:24:50.011 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52801.log.gz 2023-10-21T05:24:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42776.log 2023-10-21T05:24:50.011 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50470.log.gz 2023-10-21T05:24:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53641.log 2023-10-21T05:24:50.012 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41841.log.gz 2023-10-21T05:24:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40958.log 2023-10-21T05:24:50.012 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42776.log.gz 2023-10-21T05:24:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51411.log 2023-10-21T05:24:50.013 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53641.log.gz 2023-10-21T05:24:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44433.log 2023-10-21T05:24:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42281.log 2023-10-21T05:24:50.014 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51411.log.gz 2023-10-21T05:24:50.014 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42642.log 2023-10-21T05:24:50.014 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44433.log.gz 2023-10-21T05:24:50.015 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39177.log 2023-10-21T05:24:50.015 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42281.log.gz 2023-10-21T05:24:50.015 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40617.log 2023-10-21T05:24:50.015 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42642.log.gz 2023-10-21T05:24:50.016 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39249.log 2023-10-21T05:24:50.016 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39177.log.gz 2023-10-21T05:24:50.016 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51261.log 2023-10-21T05:24:50.016 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40617.log.gz 2023-10-21T05:24:50.017 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41979.log 2023-10-21T05:24:50.017 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39249.log.gz 2023-10-21T05:24:50.017 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52236.log 2023-10-21T05:24:50.018 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51261.log.gz 2023-10-21T05:24:50.018 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44066.log 2023-10-21T05:24:50.018 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41979.log.gz 2023-10-21T05:24:50.018 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47177.log 2023-10-21T05:24:50.019 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52236.log.gz 2023-10-21T05:24:50.019 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40094.log 2023-10-21T05:24:50.019 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44066.log.gz 2023-10-21T05:24:50.020 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50290.log 2023-10-21T05:24:50.020 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47177.log.gz 2023-10-21T05:24:50.020 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52574.log 2023-10-21T05:24:50.020 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40094.log.gz 2023-10-21T05:24:50.021 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41883.log 2023-10-21T05:24:50.021 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50290.log.gz 2023-10-21T05:24:50.021 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53094.log 2023-10-21T05:24:50.021 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52574.log.gz 2023-10-21T05:24:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40796.log 2023-10-21T05:24:50.022 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41883.log.gz 2023-10-21T05:24:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44833.log 2023-10-21T05:24:50.023 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53094.log.gz 2023-10-21T05:24:50.023 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51171.log 2023-10-21T05:24:50.023 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40796.log.gz 2023-10-21T05:24:50.023 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40887.log 2023-10-21T05:24:50.024 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44833.log.gz 2023-10-21T05:24:50.024 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47267.log 2023-10-21T05:24:50.024 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51171.log.gz 2023-10-21T05:24:50.024 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49954.log 2023-10-21T05:24:50.025 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40887.log.gz 2023-10-21T05:24:50.025 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40037.log 2023-10-21T05:24:50.025 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47267.log.gz 2023-10-21T05:24:50.026 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53824.log 2023-10-21T05:24:50.026 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49954.log.gz 2023-10-21T05:24:50.026 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42948.log 2023-10-21T05:24:50.026 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40037.log.gz 2023-10-21T05:24:50.027 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46328.log 2023-10-21T05:24:50.027 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53824.log.gz 2023-10-21T05:24:50.027 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47940.log 2023-10-21T05:24:50.027 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42948.log.gz 2023-10-21T05:24:50.028 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54628.log 2023-10-21T05:24:50.028 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46328.log.gz 2023-10-21T05:24:50.028 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45599.log 2023-10-21T05:24:50.029 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47940.log.gz 2023-10-21T05:24:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48181.log 2023-10-21T05:24:50.029 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41121.log.gz 2023-10-21T05:24:50.029 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54628.log.gz 2023-10-21T05:24:50.030 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40958.log.gz 2023-10-21T05:24:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45599.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-21T05:24:50.030 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45599.log.gz 2023-10-21T05:24:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41621.log 2023-10-21T05:24:50.031 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48181.log.gz 2023-10-21T05:24:50.031 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53613.log 2023-10-21T05:24:50.031 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44590.log 2023-10-21T05:24:50.032 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41621.log.gz 2023-10-21T05:24:50.032 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53613.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53613.log.gz -5 2023-10-21T05:24:50.032 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41453.log 2023-10-21T05:24:50.032 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52783.log 2023-10-21T05:24:50.033 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44590.log.gz 2023-10-21T05:24:50.033 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53963.log 2023-10-21T05:24:50.033 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41453.log.gz 2023-10-21T05:24:50.033 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50077.log 2023-10-21T05:24:50.034 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52783.log.gz 2023-10-21T05:24:50.034 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45448.log 2023-10-21T05:24:50.034 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53963.log.gz 2023-10-21T05:24:50.034 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51594.log 2023-10-21T05:24:50.035 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50077.log.gz 2023-10-21T05:24:50.035 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42031.log 2023-10-21T05:24:50.035 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45448.log.gz 2023-10-21T05:24:50.036 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49410.log 2023-10-21T05:24:50.036 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51594.log.gz 2023-10-21T05:24:50.036 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40509.log 2023-10-21T05:24:50.036 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42031.log.gz 2023-10-21T05:24:50.037 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40401.log 2023-10-21T05:24:50.037 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49410.log.gz 2023-10-21T05:24:50.037 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40293.log 2023-10-21T05:24:50.037 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40509.log.gz 2023-10-21T05:24:50.038 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54787.log 2023-10-21T05:24:50.038 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40401.log.gz 2023-10-21T05:24:50.038 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40563.log 2023-10-21T05:24:50.038 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40293.log.gz 2023-10-21T05:24:50.039 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39038.log 2023-10-21T05:24:50.039 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54787.log.gz 2023-10-21T05:24:50.039 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40779.log 2023-10-21T05:24:50.040 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40563.log.gz 2023-10-21T05:24:50.040 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47730.log 2023-10-21T05:24:50.040 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39038.log.gz 2023-10-21T05:24:50.040 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47880.log 2023-10-21T05:24:50.041 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40779.log.gz 2023-10-21T05:24:50.041 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47730.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.49037.log -- replaced with /var/log/ceph/ceph-client.admin.47730.log.gz 2023-10-21T05:24:50.041 INFO:teuthology.orchestra.run.smithi119.stderr: 2023-10-21T05:24:50.041 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47880.log.gz 2023-10-21T05:24:50.042 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40381.log 2023-10-21T05:24:50.042 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53378.log 2023-10-21T05:24:50.042 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49037.log.gz 2023-10-21T05:24:50.043 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40381.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40381.log.gz 2023-10-21T05:24:50.043 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/ceph-client.admin.47637.log 2023-10-21T05:24:50.043 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54360.log 2023-10-21T05:24:50.043 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53378.log.gz 2023-10-21T05:24:50.044 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47574.log 2023-10-21T05:24:50.044 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47637.log.gz 2023-10-21T05:24:50.044 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46997.log 2023-10-21T05:24:50.045 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54360.log.gz 2023-10-21T05:24:50.045 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40995.log 2023-10-21T05:24:50.045 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47574.log.gz 2023-10-21T05:24:50.045 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46997.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-21T05:24:50.046 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46997.log.gz 2023-10-21T05:24:50.046 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49379.log 2023-10-21T05:24:50.046 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40995.log.gz 2023-10-21T05:24:50.046 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52604.log 2023-10-21T05:24:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44710.log 2023-10-21T05:24:50.047 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49379.log.gz 2023-10-21T05:24:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43071.log 2023-10-21T05:24:50.047 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52604.log.gz 2023-10-21T05:24:50.048 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40651.log 2023-10-21T05:24:50.048 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44710.log.gz 2023-10-21T05:24:50.048 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40705.log 2023-10-21T05:24:50.048 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43071.log.gz 2023-10-21T05:24:50.049 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40472.log 2023-10-21T05:24:50.049 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40651.log.gz 2023-10-21T05:24:50.049 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42178.log 2023-10-21T05:24:50.050 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40705.log.gz 2023-10-21T05:24:50.050 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42581.log 2023-10-21T05:24:50.050 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40472.log.gz 2023-10-21T05:24:50.050 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53904.log 2023-10-21T05:24:50.051 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42178.log.gz 2023-10-21T05:24:50.051 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49620.log 2023-10-21T05:24:50.051 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42581.log.gz 2023-10-21T05:24:50.051 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51201.log 2023-10-21T05:24:50.052 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53904.log.gz 2023-10-21T05:24:50.052 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44650.log 2023-10-21T05:24:50.052 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49620.log.gz 2023-10-21T05:24:50.052 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48766.log 2023-10-21T05:24:50.053 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51201.log.gz 2023-10-21T05:24:50.053 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49165.log 2023-10-21T05:24:50.053 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44650.log.gz 2023-10-21T05:24:50.054 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40975.log 2023-10-21T05:24:50.054 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48766.log.gz 2023-10-21T05:24:50.054 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54254.log 2023-10-21T05:24:50.054 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49165.log.gz 2023-10-21T05:24:50.055 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52115.log 2023-10-21T05:24:50.055 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40975.log.gz 2023-10-21T05:24:50.055 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46208.log 2023-10-21T05:24:50.057 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54254.log.gz 2023-10-21T05:24:50.057 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50988.log 2023-10-21T05:24:50.057 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52115.log.gz 2023-10-21T05:24:50.058 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51351.log 2023-10-21T05:24:50.058 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46208.log.gz 2023-10-21T05:24:50.058 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40813.log 2023-10-21T05:24:50.058 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50988.log.gz 2023-10-21T05:24:50.059 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52206.log 2023-10-21T05:24:50.059 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51351.log.gz 2023-10-21T05:24:50.059 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41411.log 2023-10-21T05:24:50.060 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40813.log.gz 2023-10-21T05:24:50.060 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42472.log 2023-10-21T05:24:50.060 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52206.log.gz 2023-10-21T05:24:50.060 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42612.log 2023-10-21T05:24:50.061 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41411.log.gz 2023-10-21T05:24:50.061 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53225.log 2023-10-21T05:24:50.061 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42472.log.gz 2023-10-21T05:24:50.061 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46449.log 2023-10-21T05:24:50.062 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42612.log.gz 2023-10-21T05:24:50.062 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50867.log 2023-10-21T05:24:50.062 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53225.log.gz 2023-10-21T05:24:50.063 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52449.log 2023-10-21T05:24:50.063 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46449.log.gz 2023-10-21T05:24:50.063 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46877.log 2023-10-21T05:24:50.063 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50867.log.gz 2023-10-21T05:24:50.064 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48676.log 2023-10-21T05:24:50.064 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52449.log.gz 2023-10-21T05:24:50.064 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53746.log 2023-10-21T05:24:50.064 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46877.log.gz 2023-10-21T05:24:50.065 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41029.log 2023-10-21T05:24:50.065 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48676.log.gz 2023-10-21T05:24:50.065 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41495.log 2023-10-21T05:24:50.065 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53746.log.gz 2023-10-21T05:24:50.066 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41029.log.gz 2023-10-21T05:24:50.066 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49862.log 2023-10-21T05:24:50.066 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44538.log 2023-10-21T05:24:50.067 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49862.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.50410.log -- replaced with /var/log/ceph/ceph-client.admin.49862.log.gz 2023-10-21T05:24:50.067 INFO:teuthology.orchestra.run.smithi119.stderr: 2023-10-21T05:24:50.067 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45142.log 2023-10-21T05:24:50.067 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44538.log.gz 2023-10-21T05:24:50.068 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44189.log 2023-10-21T05:24:50.068 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50410.log.gz 2023-10-21T05:24:50.068 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53188.log 2023-10-21T05:24:50.068 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45142.log.gz 2023-10-21T05:24:50.069 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43913.log 2023-10-21T05:24:50.069 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44189.log.gz 2023-10-21T05:24:50.069 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.53188.log: -- /var/log/ceph/ceph-client.admin.43821.log 2023-10-21T05:24:50.070 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53188.log.gz 2023-10-21T05:24:50.070 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52513.log 2023-10-21T05:24:50.070 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43913.log.gz 2023-10-21T05:24:50.070 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54733.log 2023-10-21T05:24:50.071 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43821.log.gz 2023-10-21T05:24:50.071 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53876.log 2023-10-21T05:24:50.071 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52513.log.gz 2023-10-21T05:24:50.071 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50712.log 2023-10-21T05:24:50.072 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54733.log.gz 2023-10-21T05:24:50.072 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51080.log 2023-10-21T05:24:50.072 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53876.log.gz 2023-10-21T05:24:50.072 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44219.log 2023-10-21T05:24:50.073 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50712.log.gz 2023-10-21T05:24:50.073 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52054.log 2023-10-21T05:24:50.073 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51080.log.gz 2023-10-21T05:24:50.074 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44219.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.44219.log.gz 2023-10-21T05:24:50.074 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/ceph-client.admin.34609.log 2023-10-21T05:24:50.074 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52054.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.46509.log 2023-10-21T05:24:50.074 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52054.log.gz 2023-10-21T05:24:50.075 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.34609.log: /var/log/ceph/ceph-client.admin.41915.log 2023-10-21T05:24:50.075 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34609.log.gz 2023-10-21T05:24:50.075 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46509.log.gz 2023-10-21T05:24:50.075 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40165.log 2023-10-21T05:24:50.076 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46661.log 2023-10-21T05:24:50.076 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41915.log.gz 2023-10-21T05:24:50.076 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41579.log 2023-10-21T05:24:50.077 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40165.log.gz 2023-10-21T05:24:50.077 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51231.log 2023-10-21T05:24:50.077 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46661.log.gz 2023-10-21T05:24:50.077 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41579.log.gz 2023-10-21T05:24:50.078 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46723.log 2023-10-21T05:24:50.078 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41049.log 2023-10-21T05:24:50.078 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51231.log.gz 2023-10-21T05:24:50.078 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41012.log 2023-10-21T05:24:50.079 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46723.log.gz 2023-10-21T05:24:50.079 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42551.log 2023-10-21T05:24:50.079 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41049.log.gz 2023-10-21T05:24:50.080 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39093.log 2023-10-21T05:24:50.080 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41012.log.gz 2023-10-21T05:24:50.080 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53934.log 2023-10-21T05:24:50.080 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42551.log.gz 2023-10-21T05:24:50.081 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45749.log 2023-10-21T05:24:50.081 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39093.log.gz 2023-10-21T05:24:50.081 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49650.log 2023-10-21T05:24:50.081 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53934.log.gz 2023-10-21T05:24:50.082 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54015.log 2023-10-21T05:24:50.082 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45749.log.gz 2023-10-21T05:24:50.082 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45242.log 2023-10-21T05:24:50.083 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49650.log.gz 2023-10-21T05:24:50.083 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39457.log 2023-10-21T05:24:50.083 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54015.log.gz 2023-10-21T05:24:50.083 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49006.log 2023-10-21T05:24:50.084 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45242.log.gz 2023-10-21T05:24:50.084 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49893.log 2023-10-21T05:24:50.084 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39457.log.gz 2023-10-21T05:24:50.084 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49470.log 2023-10-21T05:24:50.085 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49006.log.gz 2023-10-21T05:24:50.085 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54067.log 2023-10-21T05:24:50.085 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49893.log.gz 2023-10-21T05:24:50.086 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48976.log 2023-10-21T05:24:50.086 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49470.log.gz 2023-10-21T05:24:50.086 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45112.log 2023-10-21T05:24:50.086 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54067.log.gz 2023-10-21T05:24:50.087 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52419.log 2023-10-21T05:24:50.087 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48976.log.gz 2023-10-21T05:24:50.087 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48826.log 2023-10-21T05:24:50.087 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45112.log.gz 2023-10-21T05:24:50.088 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49985.log 2023-10-21T05:24:50.088 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52419.log.gz 2023-10-21T05:24:50.088 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42338.log 2023-10-21T05:24:50.089 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48826.log.gz 2023-10-21T05:24:50.089 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43793.log 2023-10-21T05:24:50.089 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49985.log.gz 2023-10-21T05:24:50.089 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48462.log 2023-10-21T05:24:50.090 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42338.log.gz 2023-10-21T05:24:50.090 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34633.log 2023-10-21T05:24:50.090 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43793.log.gz 2023-10-21T05:24:50.091 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48401.log 2023-10-21T05:24:50.091 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48462.log.gz 2023-10-21T05:24:50.091 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.34633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49710.log 2023-10-21T05:24:50.091 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34633.log.gz 2023-10-21T05:24:50.092 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39178.log 2023-10-21T05:24:50.092 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48401.log.gz 2023-10-21T05:24:50.092 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51321.log 2023-10-21T05:24:50.092 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49710.log.gz 2023-10-21T05:24:50.093 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48736.log 2023-10-21T05:24:50.093 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39178.log.gz 2023-10-21T05:24:50.093 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53300.log 2023-10-21T05:24:50.093 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51321.log.gz 2023-10-21T05:24:50.094 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51502.log 2023-10-21T05:24:50.094 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48736.log.gz 2023-10-21T05:24:50.094 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46118.log 2023-10-21T05:24:50.094 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53300.log.gz 2023-10-21T05:24:50.095 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54681.log 2023-10-21T05:24:50.095 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51502.log.gz 2023-10-21T05:24:50.095 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43441.log 2023-10-21T05:24:50.096 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46118.log.gz 2023-10-21T05:24:50.096 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47668.log 2023-10-21T05:24:50.096 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54681.log.gz 2023-10-21T05:24:50.096 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47027.log 2023-10-21T05:24:50.097 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43441.log.gz 2023-10-21T05:24:50.097 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48796.log 2023-10-21T05:24:50.097 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47668.log.gz 2023-10-21T05:24:50.098 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51841.log 2023-10-21T05:24:50.098 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47027.log.gz 2023-10-21T05:24:50.098 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49802.log 2023-10-21T05:24:50.098 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48796.log.gz 2023-10-21T05:24:50.099 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41169.log 2023-10-21T05:24:50.099 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51841.log.gz 2023-10-21T05:24:50.099 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45357.log 2023-10-21T05:24:50.100 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49802.log.gz 2023-10-21T05:24:50.100 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40904.log 2023-10-21T05:24:50.100 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41169.log.gz 2023-10-21T05:24:50.100 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50107.log 2023-10-21T05:24:50.101 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45357.log.gz 2023-10-21T05:24:50.101 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49500.log 2023-10-21T05:24:50.101 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40904.log.gz 2023-10-21T05:24:50.101 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51441.log 2023-10-21T05:24:50.102 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50107.log.gz 2023-10-21T05:24:50.102 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48151.log 2023-10-21T05:24:50.102 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49500.log.gz 2023-10-21T05:24:50.103 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43192.log 2023-10-21T05:24:50.103 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51441.log.gz 2023-10-21T05:24:50.103 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47237.log 2023-10-21T05:24:50.103 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48151.log.gz 2023-10-21T05:24:50.104 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45845.log 2023-10-21T05:24:50.104 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43192.log.gz 2023-10-21T05:24:50.104 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42227.log 2023-10-21T05:24:50.104 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47237.log.gz 2023-10-21T05:24:50.105 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41705.log 2023-10-21T05:24:50.105 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45845.log.gz 2023-10-21T05:24:50.105 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42227.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-21T05:24:50.105 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42227.log.gz 2023-10-21T05:24:50.106 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53075.log 2023-10-21T05:24:50.106 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41705.log.gz 2023-10-21T05:24:50.106 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39482.log 2023-10-21T05:24:50.107 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41789.log 2023-10-21T05:24:50.107 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53075.log.gz 2023-10-21T05:24:50.107 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39482.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.39482.log.gz /var/log/ceph/ceph.tmp-client.admin.30924.log 2023-10-21T05:24:50.107 INFO:teuthology.orchestra.run.smithi119.stderr: 2023-10-21T05:24:50.108 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41369.log 2023-10-21T05:24:50.108 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41789.log.gz 2023-10-21T05:24:50.108 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph.tmp-client.admin.30924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2023-10-21T05:24:50.108 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30924.log.gz 2023-10-21T05:24:50.109 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39256.log 2023-10-21T05:24:50.109 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41369.log.gz 2023-10-21T05:24:50.109 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54497.log 2023-10-21T05:24:50.109 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2023-10-21T05:24:50.110 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46238.log 2023-10-21T05:24:50.110 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39256.log.gz 2023-10-21T05:24:50.110 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50590.log 2023-10-21T05:24:50.111 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54497.log.gz 2023-10-21T05:24:50.111 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46238.log.gz 2023-10-21T05:24:50.111 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40239.log 2023-10-21T05:24:50.111 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47328.log 2023-10-21T05:24:50.112 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50590.log.gz 2023-10-21T05:24:50.112 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51963.log 2023-10-21T05:24:50.112 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40239.log.gz 2023-10-21T05:24:50.112 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51716.log 2023-10-21T05:24:50.113 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47328.log.gz 2023-10-21T05:24:50.113 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51963.log.gzgzip 2023-10-21T05:24:50.113 INFO:teuthology.orchestra.run.smithi119.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44958.log 2023-10-21T05:24:50.114 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52145.log 2023-10-21T05:24:50.114 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51716.log.gz 2023-10-21T05:24:50.114 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43288.log 2023-10-21T05:24:50.114 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44958.log.gz 2023-10-21T05:24:50.115 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53720.log 2023-10-21T05:24:50.115 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52145.log.gz 2023-10-21T05:24:50.115 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53560.log 2023-10-21T05:24:50.115 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43288.log.gz 2023-10-21T05:24:50.116 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48432.log 2023-10-21T05:24:50.116 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53720.log.gz 2023-10-21T05:24:50.116 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42148.log 2023-10-21T05:24:50.117 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53560.log.gz 2023-10-21T05:24:50.117 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51780.log 2023-10-21T05:24:50.117 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48432.log.gz 2023-10-21T05:24:50.117 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54813.log 2023-10-21T05:24:50.118 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42148.log.gz 2023-10-21T05:24:50.118 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47057.log 2023-10-21T05:24:50.118 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51780.log.gz 2023-10-21T05:24:50.118 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41201.log 2023-10-21T05:24:50.119 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54813.log.gz 2023-10-21T05:24:50.119 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48341.log 2023-10-21T05:24:50.119 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47057.log.gz 2023-10-21T05:24:50.119 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54307.log 2023-10-21T05:24:50.120 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41201.log.gz 2023-10-21T05:24:50.120 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41495.log.gz 2023-10-21T05:24:50.120 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46600.log/var/log/ceph/ceph-client.admin.48341.log: 2023-10-21T05:24:50.121 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48341.log.gz 2023-10-21T05:24:50.121 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50928.log 2023-10-21T05:24:50.121 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54307.log.gz 2023-10-21T05:24:50.121 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39064.log 2023-10-21T05:24:50.122 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46600.log.gz 2023-10-21T05:24:50.122 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50806.log 2023-10-21T05:24:50.122 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50928.log.gz 2023-10-21T05:24:50.123 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52544.log 2023-10-21T05:24:50.123 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39064.log.gz 2023-10-21T05:24:50.123 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44928.log 2023-10-21T05:24:50.123 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50806.log.gz 2023-10-21T05:24:50.124 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40850.log 2023-10-21T05:24:50.124 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52544.log.gz 2023-10-21T05:24:50.124 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54863.log 2023-10-21T05:24:50.124 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44928.log.gz 2023-10-21T05:24:50.125 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47970.log 2023-10-21T05:24:50.125 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40850.log.gz 2023-10-21T05:24:50.125 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44563.log 2023-10-21T05:24:50.125 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54863.log.gz 2023-10-21T05:24:50.126 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42827.log 2023-10-21T05:24:50.126 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47970.log.gz 2023-10-21T05:24:50.126 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43741.log 2023-10-21T05:24:50.127 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44563.log.gz 2023-10-21T05:24:50.127 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49740.log 2023-10-21T05:24:50.127 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42827.log.gz 2023-10-21T05:24:50.127 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52482.log 2023-10-21T05:24:50.128 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43741.log.gz 2023-10-21T05:24:50.128 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44372.log 2023-10-21T05:24:50.128 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49740.log.gz 2023-10-21T05:24:50.128 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39257.log 2023-10-21T05:24:50.129 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52482.log.gz 2023-10-21T05:24:50.129 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50772.log 2023-10-21T05:24:50.129 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44372.log.gz 2023-10-21T05:24:50.129 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39179.log 2023-10-21T05:24:50.130 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39257.log.gz 2023-10-21T05:24:50.130 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53798.log 2023-10-21T05:24:50.130 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50772.log.gz 2023-10-21T05:24:50.131 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40489.log 2023-10-21T05:24:50.131 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39179.log.gz 2023-10-21T05:24:50.131 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53798.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-21T05:24:50.131 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53798.log.gz 2023-10-21T05:24:50.132 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45968.log 2023-10-21T05:24:50.132 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40489.log.gz 2023-10-21T05:24:50.132 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46268.log 2023-10-21T05:24:50.132 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45937.log 2023-10-21T05:24:50.133 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45968.log.gz 2023-10-21T05:24:50.133 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44898.log 2023-10-21T05:24:50.133 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46268.log.gz 2023-10-21T05:24:50.133 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41233.log 2023-10-21T05:24:50.134 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45937.log.gz 2023-10-21T05:24:50.134 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50682.log 2023-10-21T05:24:50.134 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44898.log.gz 2023-10-21T05:24:50.135 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41233.log.gz 2023-10-21T05:24:50.135 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49286.log 2023-10-21T05:24:50.135 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48310.log 2023-10-21T05:24:50.135 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50682.log.gz 2023-10-21T05:24:50.136 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44782.log 2023-10-21T05:24:50.136 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49286.log.gz 2023-10-21T05:24:50.136 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48886.log 2023-10-21T05:24:50.136 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48310.log.gz 2023-10-21T05:24:50.137 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42398.log 2023-10-21T05:24:50.137 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44782.log.gz 2023-10-21T05:24:50.137 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52023.log 2023-10-21T05:24:50.138 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48886.log.gz 2023-10-21T05:24:50.138 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54576.log 2023-10-21T05:24:50.138 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42398.log.gz 2023-10-21T05:24:50.138 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52023.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-21T05:24:50.139 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52023.log.gz 2023-10-21T05:24:50.139 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51871.log 2023-10-21T05:24:50.139 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54576.log.gz 2023-10-21T05:24:50.139 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48120.log 2023-10-21T05:24:50.140 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48946.log 2023-10-21T05:24:50.140 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51871.log.gz 2023-10-21T05:24:50.140 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52357.log 2023-10-21T05:24:50.141 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48120.log.gz 2023-10-21T05:24:50.141 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48586.log 2023-10-21T05:24:50.141 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48946.log.gz 2023-10-21T05:24:50.141 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43513.log 2023-10-21T05:24:50.142 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52357.log.gz 2023-10-21T05:24:50.142 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48586.log.gz 2023-10-21T05:24:50.142 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40526.log 2023-10-21T05:24:50.142 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43513.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.46479.log -- replaced with /var/log/ceph/ceph-client.admin.43513.log.gz 2023-10-21T05:24:50.143 INFO:teuthology.orchestra.run.smithi119.stderr: 2023-10-21T05:24:50.143 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40526.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40526.log.gz -- 2023-10-21T05:24:50.143 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/ceph-client.admin.40364.log 2023-10-21T05:24:50.143 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46479.log.gz 2023-10-21T05:24:50.144 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51534.log 2023-10-21T05:24:50.144 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47790.log 2023-10-21T05:24:50.144 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40364.log.gz 2023-10-21T05:24:50.144 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51534.log.gz 2023-10-21T05:24:50.145 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51811.log 2023-10-21T05:24:50.145 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45569.log 2023-10-21T05:24:50.145 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47790.log.gz 2023-10-21T05:24:50.146 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51811.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.51811.log.gz 2023-10-21T05:24:50.146 INFO:teuthology.orchestra.run.smithi119.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50137.log 2023-10-21T05:24:50.146 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46358.log 2023-10-21T05:24:50.146 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45569.log.gz 2023-10-21T05:24:50.147 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52820.log 2023-10-21T05:24:50.147 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50137.log.gz 2023-10-21T05:24:50.147 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46937.log 2023-10-21T05:24:50.147 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46358.log.gz 2023-10-21T05:24:50.148 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39331.log 2023-10-21T05:24:50.148 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52820.log.gz 2023-10-21T05:24:50.148 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49101.log 2023-10-21T05:24:50.148 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46937.log.gz 2023-10-21T05:24:50.149 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39331.log.gz 2023-10-21T05:24:50.149 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42524.log 2023-10-21T05:24:50.149 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40867.log 2023-10-21T05:24:50.149 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49101.log.gz 2023-10-21T05:24:50.150 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42524.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46816.log 2023-10-21T05:24:50.150 INFO:teuthology.orchestra.run.smithi119.stderr: 2023-10-21T05:24:50.150 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40867.log: 90.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-21T05:24:50.151 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40867.log.gz 2023-10-21T05:24:50.151 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49832.log 2023-10-21T05:24:50.151 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2023-10-21T05:24:50.151 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46816.log.gz 2023-10-21T05:24:50.152 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34657.log 2023-10-21T05:24:50.152 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49832.log.gz 2023-10-21T05:24:50.152 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43162.log 2023-10-21T05:24:50.152 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2023-10-21T05:24:50.153 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.34657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41066.log 2023-10-21T05:24:50.153 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34657.log.gz 2023-10-21T05:24:50.153 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45276.log 2023-10-21T05:24:50.153 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43162.log.gz 2023-10-21T05:24:50.154 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44620.log 2023-10-21T05:24:50.154 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41066.log.gz 2023-10-21T05:24:50.154 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54386.log/var/log/ceph/ceph-client.admin.45276.log: 2023-10-21T05:24:50.154 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45276.log.gz 2023-10-21T05:24:50.155 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47358.log 2023-10-21T05:24:50.155 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44620.log.gz 2023-10-21T05:24:50.155 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54386.log.gz 2023-10-21T05:24:50.156 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47207.log 2023-10-21T05:24:50.156 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48856.log 2023-10-21T05:24:50.156 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47358.log.gz 2023-10-21T05:24:50.156 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39327.log 2023-10-21T05:24:50.157 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47207.log.gz 2023-10-21T05:24:50.157 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44402.log 2023-10-21T05:24:50.157 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48856.log.gz 2023-10-21T05:24:50.157 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43020.log 2023-10-21T05:24:50.158 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39327.log.gz 2023-10-21T05:24:50.158 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47418.log 2023-10-21T05:24:50.158 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44402.log.gz 2023-10-21T05:24:50.158 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52634.log 2023-10-21T05:24:50.159 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43020.log.gz 2023-10-21T05:24:50.159 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53482.log 2023-10-21T05:24:50.159 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47418.log.gz 2023-10-21T05:24:50.159 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52664.log 2023-10-21T05:24:50.160 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52634.log.gz 2023-10-21T05:24:50.160 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40003.log 2023-10-21T05:24:50.160 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53482.log.gz 2023-10-21T05:24:50.161 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52893.log 2023-10-21T05:24:50.161 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52664.log.gz 2023-10-21T05:24:50.161 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54413.log 2023-10-21T05:24:50.161 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40003.log.gz 2023-10-21T05:24:50.162 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43380.log 2023-10-21T05:24:50.162 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52893.log.gz 2023-10-21T05:24:50.162 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51933.log 2023-10-21T05:24:50.162 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54413.log.gz 2023-10-21T05:24:50.163 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43132.log 2023-10-21T05:24:50.163 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43380.log.gz 2023-10-21T05:24:50.163 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49923.log 2023-10-21T05:24:50.163 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51933.log.gz 2023-10-21T05:24:50.164 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52948.log 2023-10-21T05:24:50.164 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43132.log.gz 2023-10-21T05:24:50.164 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47850.log 2023-10-21T05:24:50.165 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49923.log.gz 2023-10-21T05:24:50.165 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5/var/log/ceph/ceph-client.admin.52948.log: --verbose -- /var/log/ceph/ceph-client.admin.46088.log 2023-10-21T05:24:50.165 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52948.log.gz 2023-10-21T05:24:50.165 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45539.log 2023-10-21T05:24:50.166 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47850.log.gz 2023-10-21T05:24:50.166 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48030.log 2023-10-21T05:24:50.166 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46088.log.gz 2023-10-21T05:24:50.166 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46539.log 2023-10-21T05:24:50.167 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45539.log.gz 2023-10-21T05:24:50.167 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48030.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.48030.log.gz 2023-10-21T05:24:50.167 INFO:teuthology.orchestra.run.smithi119.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51018.log 2023-10-21T05:24:50.168 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48090.log 2023-10-21T05:24:50.168 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46539.log.gz 2023-10-21T05:24:50.168 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45689.log 2023-10-21T05:24:50.168 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51018.log.gz 2023-10-21T05:24:50.169 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48090.log.gz 2023-10-21T05:24:50.169 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53170.log 2023-10-21T05:24:50.169 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40054.log 2023-10-21T05:24:50.169 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45689.log.gz 2023-10-21T05:24:50.170 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34515.log 2023-10-21T05:24:50.170 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53170.log.gz 2023-10-21T05:24:50.170 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43594.log 2023-10-21T05:24:50.170 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40054.log.gz 2023-10-21T05:24:50.171 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48646.log 2023-10-21T05:24:50.171 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.34515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34515.log.gz 2023-10-21T05:24:50.171 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47820.log 2023-10-21T05:24:50.172 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43594.log.gz 2023-10-21T05:24:50.172 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48646.log.gz 2023-10-21T05:24:50.172 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54839.log 2023-10-21T05:24:50.172 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43624.log 2023-10-21T05:24:50.173 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47820.log.gz 2023-10-21T05:24:50.173 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45998.log 2023-10-21T05:24:50.173 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54839.log.gz 2023-10-21T05:24:50.173 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40671.log 2023-10-21T05:24:50.174 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43624.log.gz 2023-10-21T05:24:50.174 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45304.log 2023-10-21T05:24:50.174 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45998.log.gz 2023-10-21T05:24:50.174 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45508.log 2023-10-21T05:24:50.175 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40671.log.gz 2023-10-21T05:24:50.175 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51655.log 2023-10-21T05:24:50.175 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45304.log.gz 2023-10-21T05:24:50.176 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49560.log 2023-10-21T05:24:50.176 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45508.log.gz 2023-10-21T05:24:50.176 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51993.log 2023-10-21T05:24:50.176 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51655.log.gz 2023-10-21T05:24:50.177 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51140.log 2023-10-21T05:24:50.177 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49560.log.gz 2023-10-21T05:24:50.177 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53989.log 2023-10-21T05:24:50.177 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51993.log.gz 2023-10-21T05:24:50.178 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50168.log 2023-10-21T05:24:50.178 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51140.log.gz 2023-10-21T05:24:50.178 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45813.log 2023-10-21T05:24:50.178 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53989.log.gz 2023-10-21T05:24:50.179 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45719.log 2023-10-21T05:24:50.179 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50168.log.gz 2023-10-21T05:24:50.179 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49226.log 2023-10-21T05:24:50.179 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45813.log.gz 2023-10-21T05:24:50.180 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47087.log 2023-10-21T05:24:50.180 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45719.log.gz 2023-10-21T05:24:50.180 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43973.log 2023-10-21T05:24:50.181 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49226.log.gz 2023-10-21T05:24:50.181 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39013.log 2023-10-21T05:24:50.181 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47087.log.gz 2023-10-21T05:24:50.181 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42088.log 2023-10-21T05:24:50.182 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43973.log.gz 2023-10-21T05:24:50.182 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46569.log 2023-10-21T05:24:50.182 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39013.log.gz 2023-10-21T05:24:50.182 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54228.log 2023-10-21T05:24:50.183 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42088.log.gz 2023-10-21T05:24:50.183 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47388.log 2023-10-21T05:24:50.183 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46569.log.gz 2023-10-21T05:24:50.184 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-21T05:24:50.184 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54228.log.gz 2023-10-21T05:24:50.184 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53667.log 2023-10-21T05:24:50.184 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47388.log.gz 2023-10-21T05:24:50.185 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54120.log 2023-10-21T05:24:50.185 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40543.log 2023-10-21T05:24:50.185 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53667.log.gz 2023-10-21T05:24:50.185 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40435.log 2023-10-21T05:24:50.186 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54120.log.gz 2023-10-21T05:24:50.186 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44806.log 2023-10-21T05:24:50.186 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40543.log.gz 2023-10-21T05:24:50.187 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41103.log 2023-10-21T05:24:50.187 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40435.log.gz 2023-10-21T05:24:50.187 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45659.log 2023-10-21T05:24:50.187 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44806.log.gz 2023-10-21T05:24:50.188 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45172.log 2023-10-21T05:24:50.188 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41103.log.gz 2023-10-21T05:24:50.188 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48279.log 2023-10-21T05:24:50.188 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45659.log.gz 2023-10-21T05:24:50.189 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43654.log 2023-10-21T05:24:50.189 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45172.log.gz 2023-10-21T05:24:50.189 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52326.log 2023-10-21T05:24:50.189 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48279.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48279.log.gz 2023-10-21T05:24:50.190 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52754.log 2023-10-21T05:24:50.190 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43654.log.gz 2023-10-21T05:24:50.190 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39431.log 2023-10-21T05:24:50.191 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52326.log.gz 2023-10-21T05:24:50.191 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45082.log 2023-10-21T05:24:50.191 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52754.log.gz 2023-10-21T05:24:50.191 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42058.log 2023-10-21T05:24:50.192 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39431.log.gz 2023-10-21T05:24:50.192 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40688.log 2023-10-21T05:24:50.192 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45082.log.gz 2023-10-21T05:24:50.192 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50897.log 2023-10-21T05:24:50.193 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42058.log.gz 2023-10-21T05:24:50.193 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53404.log 2023-10-21T05:24:50.193 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40688.log.gz 2023-10-21T05:24:50.194 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43044.log 2023-10-21T05:24:50.194 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50897.log.gz 2023-10-21T05:24:50.194 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43537.log 2023-10-21T05:24:50.194 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53404.log.gz 2023-10-21T05:24:50.195 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44510.log 2023-10-21T05:24:50.195 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43044.log.gz 2023-10-21T05:24:50.195 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53114.log 2023-10-21T05:24:50.195 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43537.log.gz 2023-10-21T05:24:50.196 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53352.log 2023-10-21T05:24:50.196 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44510.log.gz 2023-10-21T05:24:50.196 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53132.log 2023-10-21T05:24:50.196 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53114.log.gz 2023-10-21T05:24:50.197 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43315.log 2023-10-21T05:24:50.197 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53352.log.gz 2023-10-21T05:24:50.197 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47910.log 2023-10-21T05:24:50.198 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53132.log.gz 2023-10-21T05:24:50.198 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53508.log 2023-10-21T05:24:50.198 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43315.log.gz 2023-10-21T05:24:50.198 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51110.log 2023-10-21T05:24:50.199 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47910.log.gz 2023-10-21T05:24:50.199 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45782.log 2023-10-21T05:24:50.199 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53508.log.gz 2023-10-21T05:24:50.199 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40148.log 2023-10-21T05:24:50.200 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51110.log.gz 2023-10-21T05:24:50.200 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49440.log 2023-10-21T05:24:50.200 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45782.log.gz 2023-10-21T05:24:50.200 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47147.log 2023-10-21T05:24:50.201 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40148.log.gz 2023-10-21T05:24:50.201 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53020.log 2023-10-21T05:24:50.201 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49440.log.gz 2023-10-21T05:24:50.201 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41083.log/var/log/ceph/ceph-client.admin.47147.log: 2023-10-21T05:24:50.202 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47147.log.gz 2023-10-21T05:24:50.202 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39508.log 2023-10-21T05:24:50.202 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53020.log.gz 2023-10-21T05:24:50.203 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46388.log 2023-10-21T05:24:50.203 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41083.log.gz 2023-10-21T05:24:50.203 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52724.log 2023-10-21T05:24:50.203 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39508.log.gz 2023-10-21T05:24:50.204 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45418.log 2023-10-21T05:24:50.204 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46388.log.gz 2023-10-21T05:24:50.204 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49348.log 2023-10-21T05:24:50.204 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52724.log.gz 2023-10-21T05:24:50.205 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46847.log 2023-10-21T05:24:50.205 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45418.log.gz 2023-10-21T05:24:50.205 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51050.log 2023-10-21T05:24:50.205 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49348.log.gz 2023-10-21T05:24:50.206 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43769.log 2023-10-21T05:24:50.206 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46847.log.gz 2023-10-21T05:24:50.206 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43264.log 2023-10-21T05:24:50.207 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51050.log.gz 2023-10-21T05:24:50.207 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40347.log 2023-10-21T05:24:50.207 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43769.log.gz 2023-10-21T05:24:50.207 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46298.log 2023-10-21T05:24:50.208 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43264.log.gz 2023-10-21T05:24:50.208 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40185.log 2023-10-21T05:24:50.208 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40347.log.gz 2023-10-21T05:24:50.209 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44680.log 2023-10-21T05:24:50.209 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46298.log.gz 2023-10-21T05:24:50.209 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51902.log 2023-10-21T05:24:50.209 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40185.log.gz 2023-10-21T05:24:50.210 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40759.log 2023-10-21T05:24:50.210 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44680.log.gz 2023-10-21T05:24:50.210 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53326.log 2023-10-21T05:24:50.210 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51902.log.gz 2023-10-21T05:24:50.211 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48371.log 2023-10-21T05:24:50.211 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40759.log.gz 2023-10-21T05:24:50.211 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51624.log 2023-10-21T05:24:50.211 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53326.log.gz 2023-10-21T05:24:50.212 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51564.log 2023-10-21T05:24:50.212 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48371.log.gz 2023-10-21T05:24:50.212 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41747.log 2023-10-21T05:24:50.213 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51624.log.gz 2023-10-21T05:24:50.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53056.log 2023-10-21T05:24:50.213 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51564.log.gz 2023-10-21T05:24:50.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54707.log 2023-10-21T05:24:50.214 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41747.log.gz 2023-10-21T05:24:50.214 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44126.log 2023-10-21T05:24:50.214 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53056.log.gz 2023-10-21T05:24:50.214 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42007.log 2023-10-21T05:24:50.215 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54707.log.gz 2023-10-21T05:24:50.215 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45202.log 2023-10-21T05:24:50.215 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44126.log.gz 2023-10-21T05:24:50.216 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50837.log 2023-10-21T05:24:50.216 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42007.log.gz 2023-10-21T05:24:50.216 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50380.log 2023-10-21T05:24:50.216 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45202.log.gz 2023-10-21T05:24:50.217 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45054.log 2023-10-21T05:24:50.217 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50837.log.gz 2023-10-21T05:24:50.217 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40074.log 2023-10-21T05:24:50.217 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50380.log.gz 2023-10-21T05:24:50.218 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50440.log 2023-10-21T05:24:50.218 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45054.log.gz 2023-10-21T05:24:50.218 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42992.log 2023-10-21T05:24:50.219 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40074.log.gz 2023-10-21T05:24:50.219 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42858.log 2023-10-21T05:24:50.219 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50440.log.gz 2023-10-21T05:24:50.219 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53430.log 2023-10-21T05:24:50.220 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42992.log.gz 2023-10-21T05:24:50.220 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53772.log 2023-10-21T05:24:50.220 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42858.log.gz 2023-10-21T05:24:50.220 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49680.log 2023-10-21T05:24:50.221 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53430.log.gz 2023-10-21T05:24:50.221 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46028.log 2023-10-21T05:24:50.221 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53772.log.gz 2023-10-21T05:24:50.221 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52874.log 2023-10-21T05:24:50.222 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49680.log.gz 2023-10-21T05:24:50.222 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50198.log 2023-10-21T05:24:50.222 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46028.log.gz 2023-10-21T05:24:50.223 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52929.log 2023-10-21T05:24:50.223 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52874.log.gz 2023-10-21T05:24:50.223 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34692.log 2023-10-21T05:24:50.223 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50198.log.gz 2023-10-21T05:24:50.224 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42308.log 2023-10-21T05:24:50.224 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52929.log.gz 2023-10-21T05:24:50.224 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.34692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34692.log.gz 2023-10-21T05:24:50.224 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52694.log 2023-10-21T05:24:50.225 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44342.log 2023-10-21T05:24:50.225 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42308.log.gz 2023-10-21T05:24:50.225 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40202.log 2023-10-21T05:24:50.225 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52694.log.gz 2023-10-21T05:24:50.226 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47451.log 2023-10-21T05:24:50.226 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44342.log.gz 2023-10-21T05:24:50.226 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51291.log 2023-10-21T05:24:50.227 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40202.log.gz 2023-10-21T05:24:50.227 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44156.log 2023-10-21T05:24:50.227 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47451.log.gz 2023-10-21T05:24:50.227 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46755.log 2023-10-21T05:24:50.228 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51291.log.gz 2023-10-21T05:24:50.228 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45907.log 2023-10-21T05:24:50.228 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44156.log.gz 2023-10-21T05:24:50.228 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45629.log 2023-10-21T05:24:50.229 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46755.log.gz 2023-10-21T05:24:50.229 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48706.log 2023-10-21T05:24:50.229 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45907.log.gz 2023-10-21T05:24:50.230 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.45629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40921.log 2023-10-21T05:24:50.230 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45629.log.gz 2023-10-21T05:24:50.230 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42368.log 2023-10-21T05:24:50.230 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48706.log.gz 2023-10-21T05:24:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50530.log 2023-10-21T05:24:50.231 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40921.log.gz 2023-10-21T05:24:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39336.log 2023-10-21T05:24:50.231 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42368.log.gz 2023-10-21T05:24:50.232 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54202.log 2023-10-21T05:24:50.232 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50530.log.gz 2023-10-21T05:24:50.232 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44096.log 2023-10-21T05:24:50.232 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39336.log.gz 2023-10-21T05:24:50.233 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52984.log 2023-10-21T05:24:50.233 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54202.log.gz 2023-10-21T05:24:50.233 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43852.log 2023-10-21T05:24:50.234 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44096.log.gz 2023-10-21T05:24:50.234 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54093.log 2023-10-21T05:24:50.234 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52984.log.gz 2023-10-21T05:24:50.234 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48916.log 2023-10-21T05:24:50.235 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43852.log.gz 2023-10-21T05:24:50.235 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50560.log 2023-10-21T05:24:50.235 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54093.log.gz 2023-10-21T05:24:50.235 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48916.log.gz 2023-10-21T05:24:50.236 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41283.log 2023-10-21T05:24:50.236 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50651.log 2023-10-21T05:24:50.236 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50560.log.gz 2023-10-21T05:24:50.237 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52856.log 2023-10-21T05:24:50.237 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41283.log.gz 2023-10-21T05:24:50.237 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48249.log 2023-10-21T05:24:50.237 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50651.log.gz 2023-10-21T05:24:50.238 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43102.log 2023-10-21T05:24:50.238 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52856.log.gz 2023-10-21T05:24:50.238 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46967.log 2023-10-21T05:24:50.238 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48249.log.gz 2023-10-21T05:24:50.239 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47605.log 2023-10-21T05:24:50.239 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43102.log.gz 2023-10-21T05:24:50.239 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48211.log 2023-10-21T05:24:50.240 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46967.log.gz 2023-10-21T05:24:50.240 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53850.log 2023-10-21T05:24:50.240 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47605.log.gz 2023-10-21T05:24:50.240 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53534.log 2023-10-21T05:24:50.241 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48211.log.gz 2023-10-21T05:24:50.241 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46419.log 2023-10-21T05:24:50.241 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53850.log.gz 2023-10-21T05:24:50.241 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54602.log 2023-10-21T05:24:50.242 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53534.log.gz 2023-10-21T05:24:50.242 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51750.log 2023-10-21T05:24:50.242 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46419.log.gz 2023-10-21T05:24:50.243 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54602.log.gz 2023-10-21T05:24:50.243 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40941.log 2023-10-21T05:24:50.243 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44039.log 2023-10-21T05:24:50.243 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.51750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51750.log.gz 2023-10-21T05:24:50.244 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40219.log 2023-10-21T05:24:50.244 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40941.log.gz 2023-10-21T05:24:50.244 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44039.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44039.log.gz -5 2023-10-21T05:24:50.244 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40580.log 2023-10-21T05:24:50.245 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50500.log 2023-10-21T05:24:50.245 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40219.log.gz 2023-10-21T05:24:50.245 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43684.log 2023-10-21T05:24:50.245 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40580.log.gz 2023-10-21T05:24:50.246 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39119.log 2023-10-21T05:24:50.246 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50500.log.gz 2023-10-21T05:24:50.246 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42800.log 2023-10-21T05:24:50.247 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43684.log.gz 2023-10-21T05:24:50.247 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.39119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46786.log 2023-10-21T05:24:50.247 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39119.log.gz 2023-10-21T05:24:50.247 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50350.log 2023-10-21T05:24:50.248 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42800.log.gz 2023-10-21T05:24:50.248 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44011.log 2023-10-21T05:24:50.248 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46786.log.gz 2023-10-21T05:24:50.248 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49068.log 2023-10-21T05:24:50.249 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50350.log.gz 2023-10-21T05:24:50.249 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54041.log 2023-10-21T05:24:50.249 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44011.log.gz 2023-10-21T05:24:50.250 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41326.log 2023-10-21T05:24:50.250 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49068.log.gz 2023-10-21T05:24:50.250 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49256.log 2023-10-21T05:24:50.250 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54041.log.gz 2023-10-21T05:24:50.251 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48616.log 2023-10-21T05:24:50.251 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41326.log.gz 2023-10-21T05:24:50.251 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48526.log 2023-10-21T05:24:50.251 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49256.log.gz 2023-10-21T05:24:50.252 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48060.log 2023-10-21T05:24:50.252 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48616.log.gz 2023-10-21T05:24:50.252 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49317.log 2023-10-21T05:24:50.252 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48526.log.gz 2023-10-21T05:24:50.253 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.48060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46148.log 2023-10-21T05:24:50.253 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48060.log.gz 2023-10-21T05:24:50.253 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49317.log.gz 2023-10-21T05:24:50.254 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42428.log 2023-10-21T05:24:50.254 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.46148.log: -- /var/log/ceph/ceph-client.admin.50320.log 2023-10-21T05:24:50.254 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46148.log.gz 2023-10-21T05:24:50.254 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52085.log 2023-10-21T05:24:50.255 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42428.log.gz 2023-10-21T05:24:50.255 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.50320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53002.log 2023-10-21T05:24:50.255 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50320.log.gz 2023-10-21T05:24:50.255 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43236.log 2023-10-21T05:24:50.256 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.52085.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52085.log.gz 2023-10-21T05:24:50.256 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.53002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44464.log 2023-10-21T05:24:50.256 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53002.log.gz 2023-10-21T05:24:50.256 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40111.log 2023-10-21T05:24:50.257 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43236.log.gz 2023-10-21T05:24:50.257 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40833.log 2023-10-21T05:24:50.257 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.44464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44464.log.gz 2023-10-21T05:24:50.257 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40327.log 2023-10-21T05:24:50.258 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40111.log.gz 2023-10-21T05:24:50.258 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54468.log 2023-10-21T05:24:50.258 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40833.log.gz 2023-10-21T05:24:50.259 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42672.log 2023-10-21T05:24:50.259 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40327.log.gz 2023-10-21T05:24:50.259 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41537.log 2023-10-21T05:24:50.259 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54468.log.gz 2023-10-21T05:24:50.260 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49134.log 2023-10-21T05:24:50.260 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42672.log.gz 2023-10-21T05:24:50.260 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.41537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54334.log 2023-10-21T05:24:50.260 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.49134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46907.log 2023-10-21T05:24:50.261 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49134.log.gz 2023-10-21T05:24:50.261 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.54334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40418.log 2023-10-21T05:24:50.261 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54334.log.gz 2023-10-21T05:24:50.261 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.46907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42748.log 2023-10-21T05:24:50.262 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46907.log.gz 2023-10-21T05:24:50.262 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.40418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47543.log 2023-10-21T05:24:50.262 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40418.log.gz 2023-10-21T05:24:50.263 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.42748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42748.log.gz 2023-10-21T05:24:50.263 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.47543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47543.log.gz 2023-10-21T05:24:50.264 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41537.log.gz 2023-10-21T05:24:50.264 INFO:teuthology.orchestra.run.smithi119.stderr: 90.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-21T05:25:01.944 INFO:teuthology.orchestra.run.smithi119.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-21T05:25:06.946 INFO:teuthology.orchestra.run.smithi119.stderr: 89.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-21T05:55:20.532 INFO:teuthology.orchestra.run.smithi119.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-21T06:05:49.299 INFO:teuthology.orchestra.run.smithi119.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-21T06:05:57.938 INFO:teuthology.orchestra.run.smithi119.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-21T06:05:57.939 INFO:teuthology.orchestra.run.smithi119.stderr: 2023-10-21T06:05:57.939 INFO:teuthology.orchestra.run.smithi119.stderr:real 41m8.156s 2023-10-21T06:05:57.940 INFO:teuthology.orchestra.run.smithi119.stderr:user 39m4.004s 2023-10-21T06:05:57.940 INFO:teuthology.orchestra.run.smithi119.stderr:sys 2m3.829s 2023-10-21T06:05:57.941 INFO:tasks.ceph:Archiving logs... 2023-10-21T06:05:57.943 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433375/remote/smithi119/log 2023-10-21T06:05:57.945 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-21T06:07:45.747 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-21T06:07:45.773 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-21T06:07:45.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-21T06:07:45.820 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-21T06:07:45.830 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-21T06:07:45.830 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-21T06:07:45.979 INFO:teuthology.orchestra.run.smithi119.stdout: remote refid st t when poll reach delay offset jitter 2023-10-21T06:07:45.980 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================== 2023-10-21T06:07:45.980 INFO:teuthology.orchestra.run.smithi119.stdout:*hv01.front.sepi 44.4.53.2 2 u 82 256 377 0.198 -3.028 0.216 2023-10-21T06:07:45.980 INFO:teuthology.orchestra.run.smithi119.stdout:+hv02.front.sepi 63.231.80.2 3 u 88 256 377 0.166 +0.027 0.521 2023-10-21T06:07:45.980 INFO:teuthology.orchestra.run.smithi119.stdout:+hv03.front.sepi 74.6.168.72 3 u 91 256 377 0.162 -2.787 0.426 2023-10-21T06:07:45.981 INFO:teuthology.orchestra.run.smithi119.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-21T06:07:45.982 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-21T06:07:45.992 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-21T06:07:45.992 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-21T06:07:46.007 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-21T06:07:46.016 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-21T06:07:46.026 INFO:teuthology.task.internal:Duration was 7966.779346 seconds 2023-10-21T06:07:46.026 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-21T06:07:46.036 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-21T06:07:46.037 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-21T06:07:46.320 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-21T06:07:46.321 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2023-10-21T06:07:46.321 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-21T06:07:46.377 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-21T06:07:46.378 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-21T06:07:46.460 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-21T06:07:46.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-21T06:07:46.813 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-21T06:07:46.823 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-21T06:07:46.824 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-21T06:07:46.869 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-21T06:07:46.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-21T06:07:46.943 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2023-10-21T06:07:46.974 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-21T06:07:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T06:07:47.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-21T06:07:47.033 INFO:teuthology.task.internal:Transferring archived files... 2023-10-21T06:07:47.034 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433375/remote/smithi119 2023-10-21T06:07:47.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-21T06:07:47.086 INFO:teuthology.task.internal:Removing archive directory... 2023-10-21T06:07:47.087 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-21T06:07:47.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-21T06:07:47.142 INFO:teuthology.task.internal:Not uploading archives. 2023-10-21T06:07:47.142 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-21T06:07:47.153 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-21T06:07:47.153 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-21T06:07:47.197 INFO:teuthology.orchestra.run.smithi119.stdout: 655476 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 21 06:07 /home/ubuntu/cephtest 2023-10-21T06:07:47.197 INFO:teuthology.orchestra.run.smithi119.stdout: 793390 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 21 04:05 /home/ubuntu/cephtest/mnt.0 2023-10-21T06:07:47.198 INFO:teuthology.orchestra.run.smithi119.stdout: 793776 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 21 04:06 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-21T06:07:47.198 INFO:teuthology.orchestra.run.smithi119.stdout: 807682 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 21 04:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-21T06:07:47.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T06:07:47.199 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-21T06:07:47.200 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi119 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-21T06:07:47.207 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-21T06:07:47.218 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-21T06:07:47.230 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi119 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-21T06:07:47.230 INFO:teuthology.nuke:Checking targets against current locks 2023-10-21T06:07:47.261 DEBUG:teuthology.nuke:shortname: smithi119 2023-10-21T06:07:47.261 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-21T06:07:47.287 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433375', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-21 03:46:35.226366', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGxiak4qSskTc2wF5c0efzOeCTK+1QFx5H7sKK9+dm0dquM9GdwhHn9zL1QnphPwznKmEs7wAhHBjtwlgK4pd24='} 2023-10-21T06:07:47.335 INFO:teuthology.orchestra.console.smithi119:Power off 2023-10-21T06:07:47.336 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-21T06:07:47.358 DEBUG:teuthology.orchestra.console.smithi119:power off output: Chassis Power Control: Down/Off 2023-10-21T06:07:47.359 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:07:47.382 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2023-10-21T06:07:51.383 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:07:51.494 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2023-10-21T06:07:55.497 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:07:55.608 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2023-10-21T06:07:59.610 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:07:59.724 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off 2023-10-21T06:07:59.825 INFO:teuthology.orchestra.console.smithi119:Power off completed 2023-10-21T06:07:59.935 INFO:teuthology.run:Summary data: description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/defaults msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 7966.779345750809 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi119 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_rraja@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3cb9a7fb51ac43a1ace7d64ab7b5f5b0 status: fail success: false 2023-10-21T06:07:59.936 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T06:08:00.035 INFO:teuthology.run:FAIL