2023-10-21T03:48:31.707 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-21T03:48:31.708 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rraja@teuthology --archive /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433358 --description rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} --name rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi --verbose -- /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433358/orig.config.yaml 2023-10-21T03:48:31.739 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T03:48:31.812 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433358 branch: ajarr-wip-extra-debug-62994-oct-19 description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7433358' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: ajarr-wip-extra-debug-62994-oct-19 ceph: conf: client: rbd default data pool: datapool rbd default features: 1 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: default sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: ajarr-wip-extra-debug-62994-oct-19 sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab owner: scheduled_rraja@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1449 sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab sleep_before_teardown: 0 suite: rbd:cli suite_branch: ajarr-wip-extra-debug-62994-oct-19 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab targets: smithi133.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKCG8FDeKVCWmI0H1Beeo5dLHysF2o12xGZTrAo6X62dqtrOYeAG7KNJ4hjz6j0NnLRBw4zzerLLkY+JHjo+8w8= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create datapool 4 - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-20_15:15:00 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-21T03:48:31.813 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa; will attempt to use it 2023-10-21T03:48:31.814 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks 2023-10-21T03:48:31.814 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-21T03:48:31.816 INFO:teuthology.task.internal:Checking packages... 2023-10-21T03:48:31.840 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab' 2023-10-21T03:48:31.840 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-21T03:48:31.840 INFO:teuthology.packaging:ref: None 2023-10-21T03:48:31.841 INFO:teuthology.packaging:tag: None 2023-10-21T03:48:31.841 INFO:teuthology.packaging:branch: ajarr-wip-extra-debug-62994-oct-19 2023-10-21T03:48:31.841 INFO:teuthology.packaging:sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T03:48:31.842 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=ajarr-wip-extra-debug-62994-oct-19 2023-10-21T03:48:32.091 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T03:48:32.091 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-21T03:48:32.098 INFO:teuthology.task.internal:no buildpackages task found 2023-10-21T03:48:32.099 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-21T03:48:32.104 INFO:teuthology.task.internal:Saving configuration 2023-10-21T03:48:32.117 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-21T03:48:32.123 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-21T03:48:32.148 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433358', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-21 03:42:57.511951', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJoioRl6c+dyXK3gqFqGyOOFi8l1rN7fXYPLvA98Dvlf'} 2023-10-21T03:48:32.148 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-21T03:48:32.155 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-21T03:48:32.155 INFO:teuthology.run_tasks:Running task console_log... 2023-10-21T03:48:32.194 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff245671a60>, signals=[15]) 2023-10-21T03:48:32.196 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-21T03:48:32.202 INFO:teuthology.task.internal:Opening connections... 2023-10-21T03:48:32.202 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi133.front.sepia.ceph.com 2023-10-21T03:48:32.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:48:32.280 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-21T03:48:32.286 DEBUG:teuthology.orchestra.run.smithi133:> uname -m 2023-10-21T03:48:32.387 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2023-10-21T03:48:32.387 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2023-10-21T03:48:32.434 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-21T03:48:32.434 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="Ubuntu" 2023-10-21T03:48:32.434 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="22.04" 2023-10-21T03:48:32.435 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-21T03:48:32.435 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_CODENAME=jammy 2023-10-21T03:48:32.435 INFO:teuthology.orchestra.run.smithi133.stdout:ID=ubuntu 2023-10-21T03:48:32.435 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE=debian 2023-10-21T03:48:32.436 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-21T03:48:32.436 INFO:teuthology.orchestra.run.smithi133.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-21T03:48:32.436 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-21T03:48:32.436 INFO:teuthology.orchestra.run.smithi133.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-21T03:48:32.436 INFO:teuthology.orchestra.run.smithi133.stdout:UBUNTU_CODENAME=jammy 2023-10-21T03:48:32.437 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2023-10-21T03:48:32.458 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-21T03:48:32.467 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-21T03:48:32.472 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-21T03:48:32.473 DEBUG:teuthology.orchestra.run.smithi133:> test '!' -e /home/ubuntu/cephtest 2023-10-21T03:48:32.482 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-21T03:48:32.490 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-21T03:48:32.490 DEBUG:teuthology.orchestra.run.smithi133:> test -z $(ls -A /var/lib/ceph) 2023-10-21T03:48:32.542 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-21T03:48:32.574 INFO:teuthology.run_tasks:Running task kernel... 2023-10-21T03:48:32.589 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-21T03:48:32.589 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-21T03:48:32.590 DEBUG:teuthology.orchestra.run.smithi133:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-21T03:48:32.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T03:48:32.596 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2023-10-21T03:48:32.642 INFO:teuthology.orchestra.run.smithi133.stdout:5.15.0-56-generic 2023-10-21T03:48:32.642 INFO:teuthology.task.kernel:Running kernel on smithi133: 5.15.0-56-generic 2023-10-21T03:48:32.643 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-get clean 2023-10-21T03:48:32.813 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-get update 2023-10-21T03:48:33.049 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-21T03:48:33.098 INFO:teuthology.orchestra.run.smithi133.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-21T03:48:33.098 INFO:teuthology.orchestra.run.smithi133.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-21T03:48:33.293 INFO:teuthology.orchestra.run.smithi133.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-21T03:48:33.369 INFO:teuthology.orchestra.run.smithi133.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,103 kB] 2023-10-21T03:48:33.521 INFO:teuthology.orchestra.run.smithi133.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [515 kB] 2023-10-21T03:48:33.535 INFO:teuthology.orchestra.run.smithi133.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [239 kB] 2023-10-21T03:48:33.544 INFO:teuthology.orchestra.run.smithi133.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-21T03:48:33.545 INFO:teuthology.orchestra.run.smithi133.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-10-21T03:48:33.546 INFO:teuthology.orchestra.run.smithi133.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,036 kB] 2023-10-21T03:48:33.556 INFO:teuthology.orchestra.run.smithi133.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [896 kB] 2023-10-21T03:48:33.578 INFO:teuthology.orchestra.run.smithi133.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [167 kB] 2023-10-21T03:48:33.582 INFO:teuthology.orchestra.run.smithi133.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-10-21T03:48:33.582 INFO:teuthology.orchestra.run.smithi133.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-10-21T03:48:33.594 INFO:teuthology.orchestra.run.smithi133.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [995 kB] 2023-10-21T03:48:33.638 INFO:teuthology.orchestra.run.smithi133.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-10-21T03:48:33.641 INFO:teuthology.orchestra.run.smithi133.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-21T03:48:33.642 INFO:teuthology.orchestra.run.smithi133.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-21T03:48:33.643 INFO:teuthology.orchestra.run.smithi133.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-21T03:48:33.644 INFO:teuthology.orchestra.run.smithi133.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-21T03:48:33.668 INFO:teuthology.orchestra.run.smithi133.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-21T03:48:33.669 INFO:teuthology.orchestra.run.smithi133.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-10-21T03:48:33.670 INFO:teuthology.orchestra.run.smithi133.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-10-21T03:48:33.672 INFO:teuthology.orchestra.run.smithi133.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-21T03:48:33.711 INFO:teuthology.orchestra.run.smithi133.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-21T03:48:33.712 INFO:teuthology.orchestra.run.smithi133.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-10-21T03:48:33.713 INFO:teuthology.orchestra.run.smithi133.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-10-21T03:48:33.713 INFO:teuthology.orchestra.run.smithi133.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-21T03:48:33.714 INFO:teuthology.orchestra.run.smithi133.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-10-21T03:48:33.919 INFO:teuthology.orchestra.run.smithi133.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [353 kB] 2023-10-21T03:48:33.941 INFO:teuthology.orchestra.run.smithi133.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [180 kB] 2023-10-21T03:48:33.952 INFO:teuthology.orchestra.run.smithi133.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-21T03:48:33.953 INFO:teuthology.orchestra.run.smithi133.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,016 kB] 2023-10-21T03:48:34.037 INFO:teuthology.orchestra.run.smithi133.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-21T03:48:34.039 INFO:teuthology.orchestra.run.smithi133.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [164 kB] 2023-10-21T03:48:34.049 INFO:teuthology.orchestra.run.smithi133.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-10-21T03:48:34.049 INFO:teuthology.orchestra.run.smithi133.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-10-21T03:48:34.079 INFO:teuthology.orchestra.run.smithi133.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [793 kB] 2023-10-21T03:48:34.267 INFO:teuthology.orchestra.run.smithi133.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-21T03:48:34.651 INFO:teuthology.orchestra.run.smithi133.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-10-21T03:48:34.658 INFO:teuthology.orchestra.run.smithi133.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-21T03:48:34.673 INFO:teuthology.orchestra.run.smithi133.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-21T03:48:34.674 INFO:teuthology.orchestra.run.smithi133.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-21T03:48:34.677 INFO:teuthology.orchestra.run.smithi133.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-21T03:48:38.889 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 9,805 kB in 2s (5,472 kB/s) 2023-10-21T03:48:40.681 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2023-10-21T03:48:40.700 DEBUG:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-21T03:48:40.753 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2023-10-21T03:48:40.938 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2023-10-21T03:48:40.939 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2023-10-21T03:48:41.099 INFO:teuthology.orchestra.run.smithi133.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T03:48:41.100 INFO:teuthology.orchestra.run.smithi133.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-21T03:48:41.100 INFO:teuthology.orchestra.run.smithi133.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-21T03:48:41.100 INFO:teuthology.orchestra.run.smithi133.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-21T03:48:41.101 INFO:teuthology.orchestra.run.smithi133.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-21T03:48:41.101 INFO:teuthology.orchestra.run.smithi133.stdout: python2.7-minimal 2023-10-21T03:48:41.101 INFO:teuthology.orchestra.run.smithi133.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T03:48:41.102 INFO:teuthology.orchestra.run.smithi133.stdout:The following additional packages will be installed: 2023-10-21T03:48:41.103 INFO:teuthology.orchestra.run.smithi133.stdout: linux-generic linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-21T03:48:41.103 INFO:teuthology.orchestra.run.smithi133.stdout: linux-headers-generic linux-image-5.15.0-87-generic 2023-10-21T03:48:41.103 INFO:teuthology.orchestra.run.smithi133.stdout: linux-modules-5.15.0-87-generic linux-modules-extra-5.15.0-87-generic 2023-10-21T03:48:41.104 INFO:teuthology.orchestra.run.smithi133.stdout:Suggested packages: 2023-10-21T03:48:41.104 INFO:teuthology.orchestra.run.smithi133.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-21T03:48:41.104 INFO:teuthology.orchestra.run.smithi133.stdout:Recommended packages: 2023-10-21T03:48:41.105 INFO:teuthology.orchestra.run.smithi133.stdout: thermald 2023-10-21T03:48:41.157 INFO:teuthology.orchestra.run.smithi133.stdout:The following NEW packages will be installed: 2023-10-21T03:48:41.157 INFO:teuthology.orchestra.run.smithi133.stdout: linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-21T03:48:41.157 INFO:teuthology.orchestra.run.smithi133.stdout: linux-image-5.15.0-87-generic linux-modules-5.15.0-87-generic 2023-10-21T03:48:41.158 INFO:teuthology.orchestra.run.smithi133.stdout: linux-modules-extra-5.15.0-87-generic 2023-10-21T03:48:41.158 INFO:teuthology.orchestra.run.smithi133.stdout:The following packages will be upgraded: 2023-10-21T03:48:41.159 INFO:teuthology.orchestra.run.smithi133.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-21T03:48:41.236 INFO:teuthology.orchestra.run.smithi133.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-10-21T03:48:41.236 INFO:teuthology.orchestra.run.smithi133.stdout:Need to get 113 MB of archives. 2023-10-21T03:48:41.236 INFO:teuthology.orchestra.run.smithi133.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-21T03:48:41.236 INFO:teuthology.orchestra.run.smithi133.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-87-generic amd64 5.15.0-87.97 [22.4 MB] 2023-10-21T03:48:41.920 INFO:teuthology.orchestra.run.smithi133.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-87-generic amd64 5.15.0-87.97 [11.5 MB] 2023-10-21T03:48:42.118 INFO:teuthology.orchestra.run.smithi133.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-87-generic amd64 5.15.0-87.97 [64.0 MB] 2023-10-21T03:48:43.380 INFO:teuthology.orchestra.run.smithi133.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.87.84 [1,698 B] 2023-10-21T03:48:43.380 INFO:teuthology.orchestra.run.smithi133.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.87.84 [2,482 B] 2023-10-21T03:48:43.381 INFO:teuthology.orchestra.run.smithi133.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87 all 5.15.0-87.97 [12.3 MB] 2023-10-21T03:48:43.773 INFO:teuthology.orchestra.run.smithi133.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87-generic amd64 5.15.0-87.97 [2,888 kB] 2023-10-21T03:48:43.872 INFO:teuthology.orchestra.run.smithi133.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.87.84 [2,322 B] 2023-10-21T03:48:44.261 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 113 MB in 3s (41.8 MB/s) 2023-10-21T03:48:44.468 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-modules-5.15.0-87-generic. 2023-10-21T03:48:46.401 INFO:teuthology.orchestra.run.smithi133.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-21T03:48:46.406 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../0-linux-modules-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-21T03:48:46.556 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:48:48.706 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-image-5.15.0-87-generic. 2023-10-21T03:48:48.728 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../1-linux-image-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-21T03:48:48.817 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:48:49.284 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-87-generic. 2023-10-21T03:48:49.305 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-21T03:48:49.518 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:48:56.123 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../3-linux-generic_5.15.0.87.84_amd64.deb ... 2023-10-21T03:48:56.226 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-21T03:48:56.524 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.87.84_amd64.deb ... 2023-10-21T03:48:56.679 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-image-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-21T03:48:56.982 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-headers-5.15.0-87. 2023-10-21T03:48:57.004 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../5-linux-headers-5.15.0-87_5.15.0-87.97_all.deb ... 2023-10-21T03:48:57.049 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-21T03:49:02.770 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-headers-5.15.0-87-generic. 2023-10-21T03:49:02.808 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../6-linux-headers-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-21T03:49:02.853 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:49:06.057 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.87.84_amd64.deb ... 2023-10-21T03:49:06.180 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-headers-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-21T03:49:06.547 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-21T03:49:06.668 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:49:06.811 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-headers-generic (5.15.0.87.84) ... 2023-10-21T03:49:06.947 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:49:08.421 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:49:10.315 INFO:teuthology.orchestra.run.smithi133.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-21T03:49:10.315 INFO:teuthology.orchestra.run.smithi133.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-21T03:49:10.316 INFO:teuthology.orchestra.run.smithi133.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-87-generic 2023-10-21T03:49:10.316 INFO:teuthology.orchestra.run.smithi133.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-87-generic 2023-10-21T03:49:10.468 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:49:11.837 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-image-generic (5.15.0.87.84) ... 2023-10-21T03:49:11.954 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-generic (5.15.0.87.84) ... 2023-10-21T03:49:12.113 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-21T03:49:12.181 INFO:teuthology.orchestra.run.smithi133.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-21T03:49:12.181 INFO:teuthology.orchestra.run.smithi133.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-87-generic 2023-10-21T03:49:26.305 INFO:teuthology.orchestra.run.smithi133.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-21T03:49:26.305 INFO:teuthology.orchestra.run.smithi133.stdout:Sourcing file `/etc/default/grub' 2023-10-21T03:49:26.316 INFO:teuthology.orchestra.run.smithi133.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-21T03:49:26.342 INFO:teuthology.orchestra.run.smithi133.stdout:Generating grub configuration file ... 2023-10-21T03:49:26.684 INFO:teuthology.orchestra.run.smithi133.stdout:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-21T03:49:26.704 INFO:teuthology.orchestra.run.smithi133.stdout:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-21T03:49:27.056 INFO:teuthology.orchestra.run.smithi133.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-21T03:49:27.062 INFO:teuthology.orchestra.run.smithi133.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-21T03:49:27.129 INFO:teuthology.orchestra.run.smithi133.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-21T03:49:27.134 INFO:teuthology.orchestra.run.smithi133.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-21T03:49:27.310 INFO:teuthology.orchestra.run.smithi133.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-21T03:49:27.311 INFO:teuthology.orchestra.run.smithi133.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-21T03:49:27.311 INFO:teuthology.orchestra.run.smithi133.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-21T03:49:27.347 INFO:teuthology.orchestra.run.smithi133.stdout:done 2023-10-21T03:49:27.809 DEBUG:teuthology.orchestra.run.smithi133:> dpkg -s linux-image-generic 2023-10-21T03:49:27.832 INFO:teuthology.orchestra.run.smithi133.stdout:Package: linux-image-generic 2023-10-21T03:49:27.832 INFO:teuthology.orchestra.run.smithi133.stdout:Status: install ok installed 2023-10-21T03:49:27.833 INFO:teuthology.orchestra.run.smithi133.stdout:Priority: optional 2023-10-21T03:49:27.833 INFO:teuthology.orchestra.run.smithi133.stdout:Section: kernel 2023-10-21T03:49:27.833 INFO:teuthology.orchestra.run.smithi133.stdout:Installed-Size: 20 2023-10-21T03:49:27.833 INFO:teuthology.orchestra.run.smithi133.stdout:Maintainer: Ubuntu Kernel Team 2023-10-21T03:49:27.833 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: amd64 2023-10-21T03:49:27.834 INFO:teuthology.orchestra.run.smithi133.stdout:Source: linux-meta 2023-10-21T03:49:27.834 INFO:teuthology.orchestra.run.smithi133.stdout:Version: 5.15.0.87.84 2023-10-21T03:49:27.834 INFO:teuthology.orchestra.run.smithi133.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-21T03:49:27.834 INFO:teuthology.orchestra.run.smithi133.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-21T03:49:27.835 INFO:teuthology.orchestra.run.smithi133.stdout:Recommends: thermald 2023-10-21T03:49:27.835 INFO:teuthology.orchestra.run.smithi133.stdout:Description: Generic Linux kernel image 2023-10-21T03:49:27.835 INFO:teuthology.orchestra.run.smithi133.stdout: This package will always depend on the latest generic kernel image 2023-10-21T03:49:27.835 INFO:teuthology.orchestra.run.smithi133.stdout: available. 2023-10-21T03:49:27.835 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-87-generic 2023-10-21T03:49:27.836 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-21T03:49:27.836 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-21T03:49:27.836 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-21T03:49:27.836 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi133.front.sepia.ceph.com, path=None, version=distro) 2023-10-21T03:49:27.837 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-get clean 2023-10-21T03:49:27.993 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-get update 2023-10-21T03:49:28.102 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-21T03:49:28.103 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-21T03:49:28.117 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-21T03:49:28.136 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-21T03:49:29.368 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2023-10-21T03:49:29.386 DEBUG:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-21T03:49:29.438 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2023-10-21T03:49:29.605 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2023-10-21T03:49:29.606 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2023-10-21T03:49:29.750 INFO:teuthology.orchestra.run.smithi133.stdout:linux-image-generic is already the newest version (5.15.0.87.84). 2023-10-21T03:49:29.750 INFO:teuthology.orchestra.run.smithi133.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T03:49:29.750 INFO:teuthology.orchestra.run.smithi133.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-21T03:49:29.750 INFO:teuthology.orchestra.run.smithi133.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-21T03:49:29.751 INFO:teuthology.orchestra.run.smithi133.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-21T03:49:29.751 INFO:teuthology.orchestra.run.smithi133.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-21T03:49:29.751 INFO:teuthology.orchestra.run.smithi133.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-21T03:49:29.752 INFO:teuthology.orchestra.run.smithi133.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T03:49:29.787 INFO:teuthology.orchestra.run.smithi133.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-10-21T03:49:29.789 DEBUG:teuthology.orchestra.run.smithi133:> dpkg -s linux-image-generic 2023-10-21T03:49:29.849 INFO:teuthology.orchestra.run.smithi133.stdout:Package: linux-image-generic 2023-10-21T03:49:29.850 INFO:teuthology.orchestra.run.smithi133.stdout:Status: install ok installed 2023-10-21T03:49:29.850 INFO:teuthology.orchestra.run.smithi133.stdout:Priority: optional 2023-10-21T03:49:29.850 INFO:teuthology.orchestra.run.smithi133.stdout:Section: kernel 2023-10-21T03:49:29.850 INFO:teuthology.orchestra.run.smithi133.stdout:Installed-Size: 20 2023-10-21T03:49:29.851 INFO:teuthology.orchestra.run.smithi133.stdout:Maintainer: Ubuntu Kernel Team 2023-10-21T03:49:29.851 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: amd64 2023-10-21T03:49:29.851 INFO:teuthology.orchestra.run.smithi133.stdout:Source: linux-meta 2023-10-21T03:49:29.851 INFO:teuthology.orchestra.run.smithi133.stdout:Version: 5.15.0.87.84 2023-10-21T03:49:29.852 INFO:teuthology.orchestra.run.smithi133.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-21T03:49:29.852 INFO:teuthology.orchestra.run.smithi133.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-21T03:49:29.852 INFO:teuthology.orchestra.run.smithi133.stdout:Recommends: thermald 2023-10-21T03:49:29.852 INFO:teuthology.orchestra.run.smithi133.stdout:Description: Generic Linux kernel image 2023-10-21T03:49:29.852 INFO:teuthology.orchestra.run.smithi133.stdout: This package will always depend on the latest generic kernel image 2023-10-21T03:49:29.853 INFO:teuthology.orchestra.run.smithi133.stdout: available. 2023-10-21T03:49:29.853 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2023-10-21T03:49:29.897 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.SpK7u6OvPC 2023-10-21T03:49:29.898 DEBUG:teuthology.orchestra.run.smithi133:> sudo cp /boot/grub/grub.cfg /tmp/tmp.SpK7u6OvPC 2023-10-21T03:49:29.953 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.SpK7u6OvPC 2023-10-21T03:49:30.084 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.SpK7u6OvPC is 10KB 2023-10-21T03:49:30.097 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.SpK7u6OvPC 2023-10-21T03:49:30.103 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-21T03:49:30.157 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T03:49:30.157 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-21T03:49:30.158 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-21T03:49:30.226 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-87-generic 2023-10-21T03:49:30.226 DEBUG:teuthology.orchestra.run.smithi133:> sudo update-grub 2023-10-21T03:49:31.377 INFO:teuthology.orchestra.run.smithi133.stderr:Sourcing file `/etc/default/grub' 2023-10-21T03:49:31.378 INFO:teuthology.orchestra.run.smithi133.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-21T03:49:31.380 INFO:teuthology.orchestra.run.smithi133.stderr:Generating grub configuration file ... 2023-10-21T03:49:31.587 INFO:teuthology.orchestra.run.smithi133.stderr:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-21T03:49:31.593 INFO:teuthology.orchestra.run.smithi133.stderr:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-21T03:49:31.896 INFO:teuthology.orchestra.run.smithi133.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-21T03:49:31.902 INFO:teuthology.orchestra.run.smithi133.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-21T03:49:31.944 INFO:teuthology.orchestra.run.smithi133.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-21T03:49:31.950 INFO:teuthology.orchestra.run.smithi133.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-21T03:49:32.113 INFO:teuthology.orchestra.run.smithi133.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-21T03:49:32.114 INFO:teuthology.orchestra.run.smithi133.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-21T03:49:32.114 INFO:teuthology.orchestra.run.smithi133.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-21T03:49:32.138 INFO:teuthology.orchestra.run.smithi133.stderr:done 2023-10-21T03:49:32.146 DEBUG:teuthology.orchestra.run.smithi133:> sudo shutdown -r now 2023-10-21T03:50:02.178 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-21T03:50:02.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-10-21T03:50:02.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:50:20.538 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-10-21T03:50:29.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-10-21T03:50:29.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:50:32.602 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-10-21T03:50:44.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-10-21T03:50:44.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:51:03.034 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-10-21T03:51:18.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-10-21T03:51:18.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:51:18.681 DEBUG:teuthology.orchestra.run.smithi133:> true 2023-10-21T03:51:19.708 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-10-21T03:51:19.708 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-87-generic"... 2023-10-21T03:51:19.709 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2023-10-21T03:51:19.714 INFO:teuthology.orchestra.run.smithi133.stdout:5.15.0-87-generic 2023-10-21T03:51:19.714 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-87-generic vs 5.15.0-87-generic 2023-10-21T03:51:19.714 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-21T03:51:19.715 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-21T03:51:20.716 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-21T03:51:20.716 DEBUG:teuthology.orchestra.run.smithi133:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-21T03:51:21.197 INFO:teuthology.orchestra.run.smithi133.stdout:ttyS1 2023-10-21T03:51:21.224 DEBUG:teuthology.parallel:result is None 2023-10-21T03:51:21.225 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-21T03:51:21.232 INFO:teuthology.task.internal:Creating test directory... 2023-10-21T03:51:21.233 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-21T03:51:21.239 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-21T03:51:21.245 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-21T03:51:21.251 INFO:teuthology.task.internal:Creating archive directory... 2023-10-21T03:51:21.252 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-21T03:51:21.376 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-21T03:51:21.386 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-21T03:51:21.387 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-21T03:51:21.422 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-21T03:51:21.430 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-21T03:51:21.432 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-21T03:51:21.439 INFO:teuthology.task.internal:Configuring sudo... 2023-10-21T03:51:21.440 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-21T03:51:21.502 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-21T03:51:21.511 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-21T03:51:21.511 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-21T03:51:21.550 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-21T03:51:21.602 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-21T03:51:21.650 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T03:51:21.651 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-21T03:51:21.717 DEBUG:teuthology.orchestra.run.smithi133:> sudo service rsyslog restart 2023-10-21T03:51:21.887 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-21T03:51:21.895 INFO:teuthology.task.internal:Starting timer... 2023-10-21T03:51:21.896 INFO:teuthology.run_tasks:Running task pcp... 2023-10-21T03:51:21.908 INFO:teuthology.run_tasks:Running task selinux... 2023-10-21T03:51:21.917 DEBUG:teuthology.task.selinux:Excluding smithi133: OS 'ubuntu' does not support SELinux 2023-10-21T03:51:21.918 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-21T03:51:21.918 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-21T03:51:21.918 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-21T03:51:21.918 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-21T03:51:21.927 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-21T03:51:21.928 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-21T03:51:21.944 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-21T03:51:21.945 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi133.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-21T03:59:01.037 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi133.front.sepia.ceph.com')] 2023-10-21T03:59:01.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-10-21T03:59:01.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T03:59:01.122 DEBUG:teuthology.orchestra.run.smithi133:> true 2023-10-21T03:59:01.238 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-10-21T03:59:01.239 INFO:teuthology.run_tasks:Running task clock... 2023-10-21T03:59:01.258 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-21T03:59:01.259 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-21T03:59:01.260 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-21T03:59:01.309 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-21T03:59:01.309 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: Command line: ntpd -gq 2023-10-21T03:59:01.310 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: ---------------------------------------------------- 2023-10-21T03:59:01.310 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: ntp-4 is maintained by Network Time Foundation, 2023-10-21T03:59:01.310 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-21T03:59:01.310 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: corporation. Support and training for ntp-4 are 2023-10-21T03:59:01.310 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: available at https://www.nwtime.org/support 2023-10-21T03:59:01.311 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: ---------------------------------------------------- 2023-10-21T03:59:01.311 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: proto: precision = 0.045 usec (-24) 2023-10-21T03:59:01.311 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: basedate set to 2022-02-04 2023-10-21T03:59:01.311 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: gps base set to 2022-02-06 (week 2196) 2023-10-21T03:59:01.312 INFO:teuthology.orchestra.run.smithi133.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-21T03:59:01.312 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-21T03:59:01.312 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: restrict ::: KOD does nothing without LIMITED. 2023-10-21T03:59:01.313 INFO:teuthology.orchestra.run.smithi133.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-21T03:59:01.314 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: Listen and drop on 0 v6wildcard [::]:123 2023-10-21T03:59:01.314 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-21T03:59:01.314 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: Listen normally on 2 lo 127.0.0.1:123 2023-10-21T03:59:01.314 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: Listen normally on 3 enp3s0f1 172.21.15.133:123 2023-10-21T03:59:01.315 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: Listen normally on 4 lo [::1]:123 2023-10-21T03:59:01.315 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8087%5]:123 2023-10-21T03:59:01.315 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:01 ntpd[27975]: Listening on routing socket on fd #22 for interface updates 2023-10-21T03:59:02.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:02.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:02.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:02.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:02.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:02.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:02.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:02.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:02.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T03:59:02.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T03:59:02.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:02.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:02.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T03:59:02.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:02 ntpd[27975]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T03:59:04.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:04 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:04.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:04 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:04.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:04 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:04.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:04 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:04.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:04 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:04.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:04 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:04.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:04 ntpd[27975]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T03:59:04.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:04 ntpd[27975]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T03:59:04.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:04 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:04.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:04 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:04.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:04 ntpd[27975]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T03:59:04.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:04 ntpd[27975]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T03:59:06.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:06.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:06.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:06.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:06.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:06.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:06.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:06.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:06.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T03:59:06.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T03:59:06.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:06.313 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:06.313 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T03:59:06.313 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T03:59:06.313 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:06.313 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:06.314 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T03:59:06.314 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:06 ntpd[27975]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T03:59:08.309 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:08 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:08.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:08 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:08.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:08 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:08.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:08 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:08.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:08 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:08.310 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:08 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:08.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:08 ntpd[27975]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-21T03:59:08.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:08 ntpd[27975]: can't open /var/log/ntpstats/rawstats.20231021: Permission denied 2023-10-21T03:59:08.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:08 ntpd[27975]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-21T03:59:08.311 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:08 ntpd[27975]: can't open /var/log/ntpstats/peerstats.20231021: Permission denied 2023-10-21T03:59:08.312 INFO:teuthology.orchestra.run.smithi133.stdout:21 Oct 03:59:08 ntpd[27975]: ntpd: time slew +0.000765 s 2023-10-21T03:59:08.312 INFO:teuthology.orchestra.run.smithi133.stdout:ntpd: time slew +0.000765s 2023-10-21T03:59:08.312 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:08 ntpd[27975]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-21T03:59:08.313 INFO:teuthology.orchestra.run.smithi133.stderr:21 Oct 03:59:08 ntpd[27975]: can't open /var/log/ntpstats/loopstats.20231021: Permission denied 2023-10-21T03:59:08.407 INFO:teuthology.orchestra.run.smithi133.stdout: remote refid st t when poll reach delay offset jitter 2023-10-21T03:59:08.407 INFO:teuthology.orchestra.run.smithi133.stdout:============================================================================== 2023-10-21T03:59:08.407 INFO:teuthology.orchestra.run.smithi133.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-21T03:59:08.407 INFO:teuthology.orchestra.run.smithi133.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-21T03:59:08.408 INFO:teuthology.orchestra.run.smithi133.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-21T03:59:08.408 INFO:teuthology.orchestra.run.smithi133.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-21T03:59:08.408 INFO:teuthology.run_tasks:Running task install... 2023-10-21T03:59:08.418 DEBUG:teuthology.task.install:project ceph 2023-10-21T03:59:08.418 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}} 2023-10-21T03:59:08.419 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'} 2023-10-21T03:59:08.419 INFO:teuthology.task.install:Using flavor: default 2023-10-21T03:59:08.425 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-21T03:59:08.425 INFO:teuthology.task.install:extra packages: [] 2023-10-21T03:59:08.426 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-key list | grep Ceph 2023-10-21T03:59:08.505 INFO:teuthology.orchestra.run.smithi133.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-21T03:59:08.534 INFO:teuthology.orchestra.run.smithi133.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-21T03:59:08.534 INFO:teuthology.orchestra.run.smithi133.stdout:uid [ unknown] Ceph.com (release key) 2023-10-21T03:59:08.535 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-21T03:59:08.535 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T03:59:08.760 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default/ 2023-10-21T03:59:08.760 INFO:teuthology.task.install.deb:Package version is 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T03:59:08.927 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T03:59:08.927 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-21T03:59:08.942 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-get update 2023-10-21T03:59:09.239 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-21T03:59:09.252 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-21T03:59:09.313 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-21T03:59:09.396 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-21T03:59:16.011 INFO:teuthology.orchestra.run.smithi133.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy InRelease 2023-10-21T03:59:22.906 INFO:teuthology.orchestra.run.smithi133.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2023-10-21T03:59:22.945 INFO:teuthology.orchestra.run.smithi133.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-21T03:59:26.431 INFO:teuthology.orchestra.run.smithi133.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2023-10-21T03:59:26.528 INFO:teuthology.orchestra.run.smithi133.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-21T03:59:26.626 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 27.9 kB in 17s (1,594 B/s) 2023-10-21T03:59:27.517 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2023-10-21T03:59:27.538 DEBUG:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6744-g7f103c7c-1jammy cephadm=18.0.0-6744-g7f103c7c-1jammy ceph-mds=18.0.0-6744-g7f103c7c-1jammy ceph-mgr=18.0.0-6744-g7f103c7c-1jammy ceph-common=18.0.0-6744-g7f103c7c-1jammy ceph-fuse=18.0.0-6744-g7f103c7c-1jammy ceph-test=18.0.0-6744-g7f103c7c-1jammy ceph-volume=18.0.0-6744-g7f103c7c-1jammy radosgw=18.0.0-6744-g7f103c7c-1jammy python3-rados=18.0.0-6744-g7f103c7c-1jammy python3-rgw=18.0.0-6744-g7f103c7c-1jammy python3-cephfs=18.0.0-6744-g7f103c7c-1jammy python3-rbd=18.0.0-6744-g7f103c7c-1jammy libcephfs2=18.0.0-6744-g7f103c7c-1jammy libcephfs-dev=18.0.0-6744-g7f103c7c-1jammy librados2=18.0.0-6744-g7f103c7c-1jammy librbd1=18.0.0-6744-g7f103c7c-1jammy rbd-fuse=18.0.0-6744-g7f103c7c-1jammy 2023-10-21T03:59:27.590 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2023-10-21T03:59:27.756 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2023-10-21T03:59:27.757 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2023-10-21T03:59:27.906 INFO:teuthology.orchestra.run.smithi133.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T03:59:27.906 INFO:teuthology.orchestra.run.smithi133.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-21T03:59:27.907 INFO:teuthology.orchestra.run.smithi133.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-21T03:59:27.907 INFO:teuthology.orchestra.run.smithi133.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-21T03:59:27.907 INFO:teuthology.orchestra.run.smithi133.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-21T03:59:27.908 INFO:teuthology.orchestra.run.smithi133.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-21T03:59:27.908 INFO:teuthology.orchestra.run.smithi133.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T03:59:27.912 INFO:teuthology.orchestra.run.smithi133.stdout:The following additional packages will be installed: 2023-10-21T03:59:27.913 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-21T03:59:27.913 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-21T03:59:27.913 INFO:teuthology.orchestra.run.smithi133.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-21T03:59:27.914 INFO:teuthology.orchestra.run.smithi133.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-21T03:59:27.914 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-21T03:59:27.915 INFO:teuthology.orchestra.run.smithi133.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-21T03:59:27.915 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-21T03:59:27.915 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-21T03:59:27.916 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-21T03:59:27.916 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-21T03:59:27.916 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-21T03:59:27.917 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-21T03:59:27.917 INFO:teuthology.orchestra.run.smithi133.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-21T03:59:27.918 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-21T03:59:27.918 INFO:teuthology.orchestra.run.smithi133.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-21T03:59:27.918 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-21T03:59:27.919 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-21T03:59:27.919 INFO:teuthology.orchestra.run.smithi133.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-21T03:59:27.919 INFO:teuthology.orchestra.run.smithi133.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-21T03:59:27.920 INFO:teuthology.orchestra.run.smithi133.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-21T03:59:27.921 INFO:teuthology.orchestra.run.smithi133.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-21T03:59:27.922 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-21T03:59:27.922 INFO:teuthology.orchestra.run.smithi133.stdout:Suggested packages: 2023-10-21T03:59:27.922 INFO:teuthology.orchestra.run.smithi133.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-21T03:59:27.923 INFO:teuthology.orchestra.run.smithi133.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-21T03:59:27.923 INFO:teuthology.orchestra.run.smithi133.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-21T03:59:27.923 INFO:teuthology.orchestra.run.smithi133.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-21T03:59:27.924 INFO:teuthology.orchestra.run.smithi133.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-21T03:59:27.924 INFO:teuthology.orchestra.run.smithi133.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-21T03:59:27.925 INFO:teuthology.orchestra.run.smithi133.stdout: python3-watchdog 2023-10-21T03:59:27.925 INFO:teuthology.orchestra.run.smithi133.stdout:Recommended packages: 2023-10-21T03:59:27.925 INFO:teuthology.orchestra.run.smithi133.stdout: btrfs-tools 2023-10-21T03:59:28.003 INFO:teuthology.orchestra.run.smithi133.stdout:The following NEW packages will be installed: 2023-10-21T03:59:28.004 INFO:teuthology.orchestra.run.smithi133.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-21T03:59:28.004 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-21T03:59:28.004 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-21T03:59:28.004 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-21T03:59:28.004 INFO:teuthology.orchestra.run.smithi133.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-21T03:59:28.005 INFO:teuthology.orchestra.run.smithi133.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-21T03:59:28.006 INFO:teuthology.orchestra.run.smithi133.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-21T03:59:28.006 INFO:teuthology.orchestra.run.smithi133.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-21T03:59:28.006 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-21T03:59:28.006 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-21T03:59:28.007 INFO:teuthology.orchestra.run.smithi133.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-21T03:59:28.007 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-21T03:59:28.007 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-21T03:59:28.007 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-21T03:59:28.007 INFO:teuthology.orchestra.run.smithi133.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-21T03:59:28.008 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-21T03:59:28.008 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-21T03:59:28.008 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-21T03:59:28.008 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-21T03:59:28.008 INFO:teuthology.orchestra.run.smithi133.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-21T03:59:28.009 INFO:teuthology.orchestra.run.smithi133.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-21T03:59:28.009 INFO:teuthology.orchestra.run.smithi133.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-21T03:59:28.009 INFO:teuthology.orchestra.run.smithi133.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-21T03:59:28.009 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-21T03:59:28.010 INFO:teuthology.orchestra.run.smithi133.stdout: socat xmlstarlet 2023-10-21T03:59:28.064 INFO:teuthology.orchestra.run.smithi133.stdout:0 upgraded, 109 newly installed, 0 to remove and 257 not upgraded. 2023-10-21T03:59:28.065 INFO:teuthology.orchestra.run.smithi133.stdout:Need to get 184 MB of archives. 2023-10-21T03:59:28.065 INFO:teuthology.orchestra.run.smithi133.stdout:After this operation, 824 MB of additional disk space will be used. 2023-10-21T03:59:28.065 INFO:teuthology.orchestra.run.smithi133.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-21T03:59:28.168 INFO:teuthology.orchestra.run.smithi133.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-21T03:59:28.176 INFO:teuthology.orchestra.run.smithi133.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6744-g7f103c7c-1jammy [3,450 kB] 2023-10-21T03:59:28.215 INFO:teuthology.orchestra.run.smithi133.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-21T03:59:28.217 INFO:teuthology.orchestra.run.smithi133.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-21T03:59:28.317 INFO:teuthology.orchestra.run.smithi133.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-21T03:59:28.365 INFO:teuthology.orchestra.run.smithi133.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-21T03:59:28.369 INFO:teuthology.orchestra.run.smithi133.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-21T03:59:28.383 INFO:teuthology.orchestra.run.smithi133.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-21T03:59:28.387 INFO:teuthology.orchestra.run.smithi133.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-21T03:59:28.387 INFO:teuthology.orchestra.run.smithi133.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-21T03:59:28.388 INFO:teuthology.orchestra.run.smithi133.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-21T03:59:28.397 INFO:teuthology.orchestra.run.smithi133.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-21T03:59:28.397 INFO:teuthology.orchestra.run.smithi133.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-21T03:59:28.398 INFO:teuthology.orchestra.run.smithi133.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-21T03:59:28.399 INFO:teuthology.orchestra.run.smithi133.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-21T03:59:28.408 INFO:teuthology.orchestra.run.smithi133.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-21T03:59:28.409 INFO:teuthology.orchestra.run.smithi133.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-21T03:59:28.409 INFO:teuthology.orchestra.run.smithi133.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-21T03:59:28.428 INFO:teuthology.orchestra.run.smithi133.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-21T03:59:28.428 INFO:teuthology.orchestra.run.smithi133.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-21T03:59:28.429 INFO:teuthology.orchestra.run.smithi133.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-21T03:59:28.431 INFO:teuthology.orchestra.run.smithi133.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-21T03:59:28.432 INFO:teuthology.orchestra.run.smithi133.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-21T03:59:28.433 INFO:teuthology.orchestra.run.smithi133.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-21T03:59:28.433 INFO:teuthology.orchestra.run.smithi133.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-21T03:59:28.448 INFO:teuthology.orchestra.run.smithi133.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-21T03:59:28.449 INFO:teuthology.orchestra.run.smithi133.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-21T03:59:28.449 INFO:teuthology.orchestra.run.smithi133.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-21T03:59:28.459 INFO:teuthology.orchestra.run.smithi133.stdout:Get:30 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6744-g7f103c7c-1jammy [3,132 kB] 2023-10-21T03:59:28.467 INFO:teuthology.orchestra.run.smithi133.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-21T03:59:28.469 INFO:teuthology.orchestra.run.smithi133.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-21T03:59:28.469 INFO:teuthology.orchestra.run.smithi133.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-21T03:59:28.474 INFO:teuthology.orchestra.run.smithi133.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-21T03:59:28.475 INFO:teuthology.orchestra.run.smithi133.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-21T03:59:28.475 INFO:teuthology.orchestra.run.smithi133.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-21T03:59:28.476 INFO:teuthology.orchestra.run.smithi133.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-21T03:59:28.487 INFO:teuthology.orchestra.run.smithi133.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-21T03:59:28.488 INFO:teuthology.orchestra.run.smithi133.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-21T03:59:28.489 INFO:teuthology.orchestra.run.smithi133.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-21T03:59:28.506 INFO:teuthology.orchestra.run.smithi133.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-21T03:59:28.507 INFO:teuthology.orchestra.run.smithi133.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-21T03:59:28.509 INFO:teuthology.orchestra.run.smithi133.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-21T03:59:28.510 INFO:teuthology.orchestra.run.smithi133.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-21T03:59:28.512 INFO:teuthology.orchestra.run.smithi133.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-21T03:59:28.566 INFO:teuthology.orchestra.run.smithi133.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-21T03:59:28.567 INFO:teuthology.orchestra.run.smithi133.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-21T03:59:28.569 INFO:teuthology.orchestra.run.smithi133.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-21T03:59:28.569 INFO:teuthology.orchestra.run.smithi133.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-21T03:59:28.569 INFO:teuthology.orchestra.run.smithi133.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-21T03:59:28.579 INFO:teuthology.orchestra.run.smithi133.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-21T03:59:28.580 INFO:teuthology.orchestra.run.smithi133.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6744-g7f103c7c-1jammy [689 kB] 2023-10-21T03:59:28.594 INFO:teuthology.orchestra.run.smithi133.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6744-g7f103c7c-1jammy [356 kB] 2023-10-21T03:59:28.596 INFO:teuthology.orchestra.run.smithi133.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-21T03:59:28.598 INFO:teuthology.orchestra.run.smithi133.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-21T03:59:28.599 INFO:teuthology.orchestra.run.smithi133.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-21T03:59:28.602 INFO:teuthology.orchestra.run.smithi133.stdout:Get:57 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6744-g7f103c7c-1jammy [32.8 kB] 2023-10-21T03:59:28.613 INFO:teuthology.orchestra.run.smithi133.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6744-g7f103c7c-1jammy [180 kB] 2023-10-21T03:59:28.614 INFO:teuthology.orchestra.run.smithi133.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-21T03:59:28.615 INFO:teuthology.orchestra.run.smithi133.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-21T03:59:28.615 INFO:teuthology.orchestra.run.smithi133.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-21T03:59:28.616 INFO:teuthology.orchestra.run.smithi133.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-21T03:59:28.616 INFO:teuthology.orchestra.run.smithi133.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-21T03:59:28.616 INFO:teuthology.orchestra.run.smithi133.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-21T03:59:28.627 INFO:teuthology.orchestra.run.smithi133.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-21T03:59:28.627 INFO:teuthology.orchestra.run.smithi133.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-21T03:59:28.630 INFO:teuthology.orchestra.run.smithi133.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-21T03:59:28.631 INFO:teuthology.orchestra.run.smithi133.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6744-g7f103c7c-1jammy [62.4 kB] 2023-10-21T03:59:28.643 INFO:teuthology.orchestra.run.smithi133.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6744-g7f103c7c-1jammy [330 kB] 2023-10-21T03:59:28.646 INFO:teuthology.orchestra.run.smithi133.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-21T03:59:28.648 INFO:teuthology.orchestra.run.smithi133.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-21T03:59:28.648 INFO:teuthology.orchestra.run.smithi133.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-21T03:59:28.651 INFO:teuthology.orchestra.run.smithi133.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-21T03:59:28.654 INFO:teuthology.orchestra.run.smithi133.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-21T03:59:28.656 INFO:teuthology.orchestra.run.smithi133.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6744-g7f103c7c-1jammy [6,221 kB] 2023-10-21T03:59:28.658 INFO:teuthology.orchestra.run.smithi133.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-21T03:59:28.661 INFO:teuthology.orchestra.run.smithi133.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-21T03:59:28.666 INFO:teuthology.orchestra.run.smithi133.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-21T03:59:28.667 INFO:teuthology.orchestra.run.smithi133.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-21T03:59:28.667 INFO:teuthology.orchestra.run.smithi133.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-21T03:59:28.686 INFO:teuthology.orchestra.run.smithi133.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-21T03:59:28.686 INFO:teuthology.orchestra.run.smithi133.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-21T03:59:28.694 INFO:teuthology.orchestra.run.smithi133.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-21T03:59:28.694 INFO:teuthology.orchestra.run.smithi133.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-21T03:59:28.694 INFO:teuthology.orchestra.run.smithi133.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-21T03:59:28.695 INFO:teuthology.orchestra.run.smithi133.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-21T03:59:28.696 INFO:teuthology.orchestra.run.smithi133.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-21T03:59:28.853 INFO:teuthology.orchestra.run.smithi133.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6744-g7f103c7c-1jammy [112 kB] 2023-10-21T03:59:28.856 INFO:teuthology.orchestra.run.smithi133.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6744-g7f103c7c-1jammy [441 kB] 2023-10-21T03:59:28.869 INFO:teuthology.orchestra.run.smithi133.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6744-g7f103c7c-1jammy [22.6 MB] 2023-10-21T03:59:29.714 INFO:teuthology.orchestra.run.smithi133.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6744-g7f103c7c-1jammy [6,852 kB] 2023-10-21T03:59:29.845 INFO:teuthology.orchestra.run.smithi133.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6744-g7f103c7c-1jammy [238 kB] 2023-10-21T03:59:29.851 INFO:teuthology.orchestra.run.smithi133.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6744-g7f103c7c-1jammy [128 kB] 2023-10-21T03:59:29.855 INFO:teuthology.orchestra.run.smithi133.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6744-g7f103c7c-1jammy [1,496 kB] 2023-10-21T03:59:29.904 INFO:teuthology.orchestra.run.smithi133.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6744-g7f103c7c-1jammy [9,863 kB] 2023-10-21T03:59:30.229 INFO:teuthology.orchestra.run.smithi133.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6744-g7f103c7c-1jammy [31.7 MB] 2023-10-21T03:59:31.369 INFO:teuthology.orchestra.run.smithi133.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6744-g7f103c7c-1jammy [14.0 kB] 2023-10-21T03:59:31.370 INFO:teuthology.orchestra.run.smithi133.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6744-g7f103c7c-1jammy [839 kB] 2023-10-21T03:59:31.386 INFO:teuthology.orchestra.run.smithi133.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6744-g7f103c7c-1jammy [2,146 kB] 2023-10-21T03:59:31.427 INFO:teuthology.orchestra.run.smithi133.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6744-g7f103c7c-1jammy [283 kB] 2023-10-21T03:59:31.427 INFO:teuthology.orchestra.run.smithi133.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6744-g7f103c7c-1jammy [138 kB] 2023-10-21T03:59:31.430 INFO:teuthology.orchestra.run.smithi133.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6744-g7f103c7c-1jammy [1,906 kB] 2023-10-21T03:59:31.450 INFO:teuthology.orchestra.run.smithi133.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6744-g7f103c7c-1jammy [8,625 kB] 2023-10-21T03:59:31.727 INFO:teuthology.orchestra.run.smithi133.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6744-g7f103c7c-1jammy [14.2 kB] 2023-10-21T03:59:31.727 INFO:teuthology.orchestra.run.smithi133.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6744-g7f103c7c-1jammy [47.8 MB] 2023-10-21T03:59:34.163 INFO:teuthology.orchestra.run.smithi133.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6744-g7f103c7c-1jammy [120 kB] 2023-10-21T03:59:34.165 INFO:teuthology.orchestra.run.smithi133.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6744-g7f103c7c-1jammy [38.3 kB] 2023-10-21T03:59:34.166 INFO:teuthology.orchestra.run.smithi133.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6744-g7f103c7c-1jammy [12.3 MB] 2023-10-21T03:59:34.456 INFO:teuthology.orchestra.run.smithi133.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6744-g7f103c7c-1jammy [91.6 kB] 2023-10-21T03:59:34.998 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 184 MB in 6s (28.5 MB/s) 2023-10-21T03:59:35.109 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-21T03:59:35.165 INFO:teuthology.orchestra.run.smithi133.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165984 files and directories currently installed.) 2023-10-21T03:59:35.170 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-21T03:59:35.242 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-21T03:59:35.528 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-21T03:59:35.558 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-21T03:59:35.595 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-21T03:59:35.923 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-21T03:59:35.950 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-21T03:59:35.990 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-21T03:59:36.359 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-21T03:59:36.386 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-21T03:59:36.426 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-21T03:59:36.787 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-21T03:59:36.815 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-21T03:59:36.854 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T03:59:37.282 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-21T03:59:37.309 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-21T03:59:37.349 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T03:59:37.702 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-21T03:59:37.728 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-21T03:59:37.768 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T03:59:38.329 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-21T03:59:38.345 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-21T03:59:38.379 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-21T03:59:38.682 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package librados2. 2023-10-21T03:59:38.709 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../008-librados2_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:38.749 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librados2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:39.201 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package librbd1. 2023-10-21T03:59:39.228 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../009-librbd1_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:39.268 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librbd1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:39.687 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libcephfs2. 2023-10-21T03:59:39.714 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:39.754 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libcephfs2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:39.998 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-rados. 2023-10-21T03:59:40.013 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:40.048 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-rados (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:40.351 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-21T03:59:40.377 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T03:59:40.417 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-ceph-argparse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:40.669 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cephfs. 2023-10-21T03:59:40.696 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:40.736 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cephfs (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:41.047 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-21T03:59:41.073 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T03:59:41.113 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:41.441 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-21T03:59:41.468 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-21T03:59:41.508 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-21T03:59:41.837 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-prettytable. 2023-10-21T03:59:41.862 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-21T03:59:41.902 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-21T03:59:42.155 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-rbd. 2023-10-21T03:59:42.181 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:42.213 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-rbd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:42.560 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-21T03:59:42.586 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-21T03:59:42.626 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-21T03:59:42.963 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package librgw2. 2023-10-21T03:59:42.989 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../019-librgw2_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:43.029 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librgw2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:43.531 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-rgw. 2023-10-21T03:59:43.546 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:43.581 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-rgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:43.892 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-21T03:59:43.919 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-21T03:59:43.959 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-21T03:59:44.253 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libradosstriper1. 2023-10-21T03:59:44.280 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:44.320 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libradosstriper1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:44.840 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-common. 2023-10-21T03:59:44.866 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:44.914 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:46.220 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-base. 2023-10-21T03:59:46.247 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:46.296 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-base (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:47.024 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-21T03:59:47.050 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-21T03:59:47.090 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-21T03:59:47.368 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cheroot. 2023-10-21T03:59:47.384 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-21T03:59:47.419 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-21T03:59:47.679 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-21T03:59:47.706 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-21T03:59:47.738 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-21T03:59:48.040 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-21T03:59:48.067 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-21T03:59:48.107 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-21T03:59:48.593 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-21T03:59:48.620 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-21T03:59:48.660 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-21T03:59:48.937 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-tempora. 2023-10-21T03:59:48.964 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-21T03:59:48.996 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-21T03:59:49.266 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-portend. 2023-10-21T03:59:49.291 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-21T03:59:49.332 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-21T03:59:49.634 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-21T03:59:49.661 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-21T03:59:49.702 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-21T03:59:50.004 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-21T03:59:50.030 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-21T03:59:50.070 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-21T03:59:50.432 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-natsort. 2023-10-21T03:59:50.459 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-21T03:59:50.499 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-21T03:59:50.819 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-logutils. 2023-10-21T03:59:50.845 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-21T03:59:50.885 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-21T03:59:51.238 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-21T03:59:51.264 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-21T03:59:51.314 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-21T03:59:51.657 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-mako. 2023-10-21T03:59:51.683 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-21T03:59:51.723 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-21T03:59:52.060 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-21T03:59:52.087 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-21T03:59:52.127 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-21T03:59:52.429 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-21T03:59:52.456 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-21T03:59:52.496 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-21T03:59:52.824 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-webob. 2023-10-21T03:59:52.850 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-21T03:59:52.890 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-21T03:59:53.218 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-waitress. 2023-10-21T03:59:53.245 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-21T03:59:53.280 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-21T03:59:53.521 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-tempita. 2023-10-21T03:59:53.538 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-21T03:59:53.579 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-21T03:59:53.990 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-paste. 2023-10-21T03:59:54.017 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-21T03:59:54.057 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-21T03:59:54.360 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-21T03:59:54.375 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-21T03:59:54.410 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-21T03:59:54.704 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-21T03:59:54.731 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-21T03:59:54.771 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-21T03:59:55.074 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-webtest. 2023-10-21T03:59:55.100 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-21T03:59:55.140 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-21T03:59:55.452 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pecan. 2023-10-21T03:59:55.478 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-21T03:59:55.518 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-21T03:59:55.898 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-certifi. 2023-10-21T03:59:55.924 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-21T03:59:55.964 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-21T03:59:56.233 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-idna. 2023-10-21T03:59:56.260 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-21T03:59:56.292 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-21T03:59:56.603 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-urllib3. 2023-10-21T03:59:56.629 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-21T03:59:56.669 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-21T03:59:57.006 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-requests. 2023-10-21T03:59:57.032 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-21T03:59:57.073 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-21T03:59:57.367 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-21T03:59:57.394 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-21T03:59:57.433 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-21T03:59:57.736 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-21T03:59:57.763 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T03:59:57.803 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:58.166 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-21T03:59:58.192 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:58.232 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:58.527 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-openssl. 2023-10-21T03:59:58.554 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-21T03:59:58.593 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-21T03:59:58.930 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr. 2023-10-21T03:59:58.956 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:58.996 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T03:59:59.516 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mon. 2023-10-21T03:59:59.543 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T03:59:59.582 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mon (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:00.411 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-osd. 2023-10-21T04:00:00.438 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:00:00.469 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-osd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:02.264 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph. 2023-10-21T04:00:02.279 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../059-ceph_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:00:02.313 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:02.641 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-fuse. 2023-10-21T04:00:02.668 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:00:02.707 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:03.094 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mds. 2023-10-21T04:00:03.121 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:00:03.161 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mds (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:03.547 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package cephadm. 2023-10-21T04:00:03.573 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../062-cephadm_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:00:03.613 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:03.901 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-21T04:00:03.926 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-21T04:00:03.958 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-21T04:00:04.294 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-21T04:00:04.310 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-21T04:00:04.344 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-21T04:00:05.332 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-babel. 2023-10-21T04:00:05.359 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-21T04:00:05.398 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-21T04:00:05.764 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-jinja2. 2023-10-21T04:00:05.790 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-21T04:00:05.885 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-21T04:00:06.238 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-21T04:00:06.266 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:00:06.305 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:06.666 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-21T04:00:06.694 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-21T04:00:06.733 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-21T04:00:07.044 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-21T04:00:07.071 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-21T04:00:07.110 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-21T04:00:07.380 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-routes. 2023-10-21T04:00:07.396 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-21T04:00:07.430 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-21T04:00:07.750 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-21T04:00:07.777 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:00:07.819 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:08.236 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-21T04:00:08.252 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-21T04:00:08.294 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-21T04:00:08.790 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-joblib. 2023-10-21T04:00:08.816 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-21T04:00:08.856 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-21T04:00:09.184 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-21T04:00:09.212 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-21T04:00:09.251 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-21T04:00:09.665 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-sklearn. 2023-10-21T04:00:09.692 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-21T04:00:09.820 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-21T04:00:10.529 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-21T04:00:10.556 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:00:10.620 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:11.531 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cachetools. 2023-10-21T04:00:11.559 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-21T04:00:11.598 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-21T04:00:11.900 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-21T04:00:11.927 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-21T04:00:11.967 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-21T04:00:12.287 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-21T04:00:12.314 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-21T04:00:12.353 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-21T04:00:12.665 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-rsa. 2023-10-21T04:00:12.692 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-21T04:00:12.731 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-21T04:00:13.001 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-google-auth. 2023-10-21T04:00:13.028 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-21T04:00:13.059 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-21T04:00:13.379 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-21T04:00:13.406 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-21T04:00:13.445 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-21T04:00:13.808 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-websocket. 2023-10-21T04:00:13.824 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-21T04:00:13.858 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-21T04:00:14.153 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-21T04:00:14.180 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-21T04:00:14.220 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-21T04:00:14.859 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-21T04:00:14.886 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:00:14.924 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:15.278 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-21T04:00:15.305 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-21T04:00:15.344 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-21T04:00:15.647 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-21T04:00:15.674 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-21T04:00:15.713 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-21T04:00:15.933 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package jq. 2023-10-21T04:00:15.948 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-21T04:00:15.991 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-21T04:00:16.243 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package socat. 2023-10-21T04:00:16.259 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-21T04:00:16.302 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-21T04:00:16.621 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package xmlstarlet. 2023-10-21T04:00:16.637 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-21T04:00:16.680 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-21T04:00:17.059 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-test. 2023-10-21T04:00:17.086 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:00:17.125 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-test (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:19.329 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-volume. 2023-10-21T04:00:19.345 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-21T04:00:19.381 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-volume (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:19.682 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-21T04:00:19.698 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:00:19.741 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libcephfs-dev (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:20.271 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package nvme-cli. 2023-10-21T04:00:20.299 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-21T04:00:20.335 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-21T04:00:20.666 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-21T04:00:20.682 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-21T04:00:20.715 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-21T04:00:21.169 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-21T04:00:21.196 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-21T04:00:21.235 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-21T04:00:21.504 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pastescript. 2023-10-21T04:00:21.520 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-21T04:00:21.554 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-21T04:00:21.832 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pluggy. 2023-10-21T04:00:21.860 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-21T04:00:21.899 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-21T04:00:22.212 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-psutil. 2023-10-21T04:00:22.240 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-21T04:00:22.279 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-21T04:00:22.615 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-py. 2023-10-21T04:00:22.642 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-21T04:00:22.681 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-21T04:00:23.026 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pygments. 2023-10-21T04:00:23.054 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-21T04:00:23.093 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-21T04:00:23.522 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-21T04:00:23.538 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-21T04:00:23.571 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-21T04:00:23.858 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-toml. 2023-10-21T04:00:23.885 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-21T04:00:23.924 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-21T04:00:24.227 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pytest. 2023-10-21T04:00:24.254 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-21T04:00:24.293 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-21T04:00:24.588 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-simplejson. 2023-10-21T04:00:24.616 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-21T04:00:24.655 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-21T04:00:24.890 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-21T04:00:24.907 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-21T04:00:24.949 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-21T04:00:25.686 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package radosgw. 2023-10-21T04:00:25.714 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../107-radosgw_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:00:25.753 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking radosgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:26.505 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package rbd-fuse. 2023-10-21T04:00:26.522 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-21T04:00:26.564 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking rbd-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:26.860 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-21T04:00:27.110 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-21T04:00:27.236 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-21T04:00:27.494 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-21T04:00:27.845 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-21T04:00:28.317 INFO:teuthology.orchestra.run.smithi133.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-21T04:00:28.413 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:29.048 INFO:teuthology.orchestra.run.smithi133.stdout:Adding system user cephadm....done 2023-10-21T04:00:29.202 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-21T04:00:29.470 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-21T04:00:29.705 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-21T04:00:29.831 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-21T04:00:30.090 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-21T04:00:30.342 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-21T04:00:30.468 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-21T04:00:30.827 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-21T04:00:31.213 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-21T04:00:31.464 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-21T04:00:31.731 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-21T04:00:31.991 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-21T04:00:32.234 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-21T04:00:32.476 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-ceph-argparse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:32.744 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-21T04:00:32.862 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-21T04:00:33.105 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-21T04:00:33.230 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-21T04:00:33.623 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-21T04:00:33.749 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-21T04:00:34.008 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-21T04:00:34.284 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-21T04:00:34.602 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-21T04:00:35.254 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-21T04:00:35.514 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-21T04:00:35.640 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-21T04:00:35.766 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-21T04:00:36.033 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-21T04:00:36.385 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-21T04:00:36.636 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-21T04:00:37.012 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-21T04:00:37.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-21T04:00:37.632 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-21T04:00:37.874 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-21T04:00:37.975 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:38.268 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-idna (3.3-1) ... 2023-10-21T04:00:38.586 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-21T04:00:39.758 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:00:39.867 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-21T04:00:40.127 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-21T04:00:40.236 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-21T04:00:40.337 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-21T04:00:40.639 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-21T04:00:40.890 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-21T04:00:41.133 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:00:41.233 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-21T04:00:41.334 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-21T04:00:41.619 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-21T04:00:41.862 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-21T04:00:42.113 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-21T04:00:42.372 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-21T04:00:42.623 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-21T04:00:42.874 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-21T04:00:42.975 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-21T04:00:43.235 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-21T04:00:43.386 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-21T04:00:43.679 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-21T04:00:44.005 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-21T04:00:44.264 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-21T04:00:44.668 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-21T04:00:44.892 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-21T04:00:45.018 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-21T04:00:45.386 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-21T04:00:45.637 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-21T04:00:45.888 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-21T04:00:46.014 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-21T04:00:46.324 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-21T04:00:46.608 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-21T04:00:46.876 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-21T04:00:47.002 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-21T04:00:47.270 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-21T04:00:47.529 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-21T04:00:47.797 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-21T04:00:48.173 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-21T04:00:48.449 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-21T04:00:48.793 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-21T04:00:48.903 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librados2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:49.004 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librgw2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:49.130 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:49.256 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libcephfs2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:49.382 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libradosstriper1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:49.601 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-21T04:00:49.870 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librbd1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:49.971 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-modules-core (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:50.097 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:50.318 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-21T04:00:50.318 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-21T04:00:50.857 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libcephfs-dev (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:51.250 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-rados (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:51.453 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-21T04:00:52.639 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-rbd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:52.756 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up rbd-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:52.857 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-rgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:52.958 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cephfs (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:53.084 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:53.652 INFO:teuthology.orchestra.run.smithi133.stdout:Adding group ceph....done 2023-10-21T04:00:54.021 INFO:teuthology.orchestra.run.smithi133.stdout:Adding system user ceph....done 2023-10-21T04:00:54.214 INFO:teuthology.orchestra.run.smithi133.stdout:Setting system user ceph properties....done 2023-10-21T04:00:54.232 INFO:teuthology.orchestra.run.smithi133.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-21T04:00:54.373 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-21T04:00:54.697 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-21T04:00:55.202 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-test (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:55.337 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up radosgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:55.942 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-21T04:00:55.943 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-21T04:00:56.414 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-base (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:56.745 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-21T04:00:57.257 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mds (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:57.493 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-21T04:00:57.493 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-21T04:00:58.017 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:58.222 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-21T04:00:58.222 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-21T04:00:58.736 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-osd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:59.036 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-21T04:00:59.037 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-21T04:00:59.554 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:59.680 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:00:59.848 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mon (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:01:00.044 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-21T04:01:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-21T04:01:00.557 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:01:00.691 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:01:00.917 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-dashboard (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:01:01.052 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-volume (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-21T04:01:01.564 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-21T04:01:02.690 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-21T04:01:03.169 INFO:teuthology.orchestra.run.smithi133.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-21T04:01:03.173 DEBUG:teuthology.parallel:result is None 2023-10-21T04:01:03.174 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:01:03.402 DEBUG:teuthology.orchestra.run.smithi133:> dpkg-query -W -f '${Version}' ceph 2023-10-21T04:01:03.422 INFO:teuthology.orchestra.run.smithi133.stdout:18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:01:03.423 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:01:03.423 INFO:teuthology.task.install:The correct ceph version 18.0.0-6744-g7f103c7c-1jammy is installed. 2023-10-21T04:01:03.424 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-21T04:01:03.425 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:03.426 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-21T04:01:03.478 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-21T04:01:03.479 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:03.480 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/daemon-helper 2023-10-21T04:01:03.533 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-21T04:01:03.588 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-21T04:01:03.589 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:03.589 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-21T04:01:03.641 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-21T04:01:03.696 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-21T04:01:03.696 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:03.697 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/stdin-killer 2023-10-21T04:01:03.749 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-21T04:01:03.804 INFO:teuthology.run_tasks:Running task ceph... 2023-10-21T04:01:03.907 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-21T04:01:03.909 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 777 /var/log/ceph 2023-10-21T04:01:03.923 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-21T04:01:03.924 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-21T04:01:03.976 INFO:tasks.ceph:Creating extra log directories... 2023-10-21T04:01:03.977 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-21T04:01:04.047 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-21T04:01:04.048 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-21T04:01:04.048 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433358', 'branch': 'ajarr-wip-extra-debug-62994-oct-19', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7433358', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'ajarr-wip-extra-debug-62994-oct-19'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'ajarr-wip-extra-debug-62994-oct-19', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1449, 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'ajarr-wip-extra-debug-62994-oct-19', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab', 'targets': {'smithi133.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKCG8FDeKVCWmI0H1Beeo5dLHysF2o12xGZTrAo6X62dqtrOYeAG7KNJ4hjz6j0NnLRBw4zzerLLkY+JHjo+8w8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create datapool 4', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '202b180cb047e798fb131047314a862593f45403', 'timestamp': '2023-10-20_15:15:00', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-21T04:01:04.049 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-21T04:01:04.082 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/run/ceph 2023-10-21T04:01:04.139 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:04.150 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/scratch_devs of=/dev/stdout 2023-10-21T04:01:04.185 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-21T04:01:04.186 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_1 2023-10-21T04:01:04.233 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-21T04:01:04.234 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T04:01:04.234 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 884 Links: 1 2023-10-21T04:01:04.234 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T04:01:04.234 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2023-10-21 03:55:44.960438555 +0000 2023-10-21T04:01:04.235 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2023-10-21 03:55:44.580441940 +0000 2023-10-21T04:01:04.235 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2023-10-21 03:55:44.580441940 +0000 2023-10-21T04:01:04.235 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2023-10-21T04:01:04.235 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-21T04:01:04.288 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2023-10-21T04:01:04.289 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2023-10-21T04:01:04.289 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000417996 s, 1.2 MB/s 2023-10-21T04:01:04.290 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-21T04:01:04.342 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_2 2023-10-21T04:01:04.389 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-21T04:01:04.389 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T04:01:04.390 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 899 Links: 1 2023-10-21T04:01:04.390 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T04:01:04.390 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2023-10-21 03:55:45.572433104 +0000 2023-10-21T04:01:04.390 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2023-10-21 03:55:45.196436453 +0000 2023-10-21T04:01:04.390 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2023-10-21 03:55:45.196436453 +0000 2023-10-21T04:01:04.391 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2023-10-21T04:01:04.391 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-21T04:01:04.444 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2023-10-21T04:01:04.444 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2023-10-21T04:01:04.444 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000365753 s, 1.4 MB/s 2023-10-21T04:01:04.445 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-21T04:01:04.495 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_3 2023-10-21T04:01:04.541 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-21T04:01:04.542 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T04:01:04.542 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 922 Links: 1 2023-10-21T04:01:04.542 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T04:01:04.542 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2023-10-21 03:55:46.168427794 +0000 2023-10-21T04:01:04.542 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2023-10-21 03:55:45.796431107 +0000 2023-10-21T04:01:04.543 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2023-10-21 03:55:45.796431107 +0000 2023-10-21T04:01:04.543 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2023-10-21T04:01:04.543 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-21T04:01:04.597 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2023-10-21T04:01:04.597 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2023-10-21T04:01:04.598 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000363886 s, 1.4 MB/s 2023-10-21T04:01:04.599 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-21T04:01:04.647 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_4 2023-10-21T04:01:04.693 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-21T04:01:04.694 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-21T04:01:04.694 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 934 Links: 1 2023-10-21T04:01:04.694 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-21T04:01:04.694 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2023-10-21 03:55:46.756422557 +0000 2023-10-21T04:01:04.694 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2023-10-21 03:55:46.400425728 +0000 2023-10-21T04:01:04.695 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2023-10-21 03:55:46.400425728 +0000 2023-10-21T04:01:04.695 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2023-10-21T04:01:04.695 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-21T04:01:04.749 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2023-10-21T04:01:04.750 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2023-10-21T04:01:04.750 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000384004 s, 1.3 MB/s 2023-10-21T04:01:04.751 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-21T04:01:04.799 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T04:01:04.799 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-21T04:01:04.800 INFO:tasks.ceph:Generating config... 2023-10-21T04:01:04.801 INFO:tasks.ceph:[client] rbd default data pool = datapool 2023-10-21T04:01:04.802 INFO:tasks.ceph:[client] rbd default features = 1 2023-10-21T04:01:04.802 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-21T04:01:04.803 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-21T04:01:04.803 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-21T04:01:04.803 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-21T04:01:04.804 INFO:tasks.ceph:[mgr] debug monc = 20 2023-10-21T04:01:04.804 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-21T04:01:04.804 INFO:tasks.ceph:[mgr] debug objecter = 20 2023-10-21T04:01:04.804 INFO:tasks.ceph:[mgr] debug rados = 20 2023-10-21T04:01:04.805 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-21T04:01:04.805 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-21T04:01:04.805 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-21T04:01:04.805 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-21T04:01:04.806 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-21T04:01:04.806 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2023-10-21T04:01:04.806 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-21T04:01:04.806 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-21T04:01:04.807 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-21T04:01:04.807 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-21T04:01:04.808 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-21T04:01:04.808 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-21T04:01:04.809 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-21T04:01:04.809 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-21T04:01:04.810 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-21T04:01:04.810 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-21T04:01:04.811 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-21T04:01:04.811 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-21T04:01:04.811 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-21T04:01:04.812 INFO:tasks.ceph:Setting up mon.a... 2023-10-21T04:01:04.813 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-21T04:01:04.872 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/ceph.keyring 2023-10-21T04:01:04.879 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-21T04:01:04.955 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-21T04:01:05.008 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.133')] 2023-10-21T04:01:05.009 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.133', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 20, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 1}, 'mon.a': {}} 2023-10-21T04:01:05.010 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:05.010 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-21T04:01:05.054 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.133 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:01:05.132 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:01:05.133 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: generated fsid f3f3d351-07e7-41e0-8ea5-8c6ad08b8ea3 2023-10-21T04:01:05.133 INFO:teuthology.orchestra.run.smithi133.stdout:setting min_mon_release = pacific 2023-10-21T04:01:05.133 INFO:teuthology.orchestra.run.smithi133.stdout:epoch 0 2023-10-21T04:01:05.133 INFO:teuthology.orchestra.run.smithi133.stdout:fsid f3f3d351-07e7-41e0-8ea5-8c6ad08b8ea3 2023-10-21T04:01:05.134 INFO:teuthology.orchestra.run.smithi133.stdout:last_changed 2023-10-21T04:01:05.131085+0000 2023-10-21T04:01:05.134 INFO:teuthology.orchestra.run.smithi133.stdout:created 2023-10-21T04:01:05.131085+0000 2023-10-21T04:01:05.134 INFO:teuthology.orchestra.run.smithi133.stdout:min_mon_release 16 (pacific) 2023-10-21T04:01:05.134 INFO:teuthology.orchestra.run.smithi133.stdout:election_strategy: 1 2023-10-21T04:01:05.134 INFO:teuthology.orchestra.run.smithi133.stdout:0: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.a 2023-10-21T04:01:05.135 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-21T04:01:05.136 DEBUG:teuthology.orchestra.run.smithi133:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-21T04:01:05.182 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f3f3d351-07e7-41e0-8ea5-8c6ad08b8ea3... 2023-10-21T04:01:05.183 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-21T04:01:05.255 INFO:teuthology.orchestra.run.smithi133.stdout:[global] 2023-10-21T04:01:05.256 INFO:teuthology.orchestra.run.smithi133.stdout: chdir = "" 2023-10-21T04:01:05.256 INFO:teuthology.orchestra.run.smithi133.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-21T04:01:05.256 INFO:teuthology.orchestra.run.smithi133.stdout: auth supported = cephx 2023-10-21T04:01:05.256 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.257 INFO:teuthology.orchestra.run.smithi133.stdout: filestore xattr use omap = true 2023-10-21T04:01:05.257 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.257 INFO:teuthology.orchestra.run.smithi133.stdout: mon clock drift allowed = 1.000 2023-10-21T04:01:05.257 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.257 INFO:teuthology.orchestra.run.smithi133.stdout: osd crush chooseleaf type = 0 2023-10-21T04:01:05.257 INFO:teuthology.orchestra.run.smithi133.stdout: auth debug = true 2023-10-21T04:01:05.258 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.258 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on old message = true 2023-10-21T04:01:05.258 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on bug = true 2023-10-21T04:01:05.258 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.258 INFO:teuthology.orchestra.run.smithi133.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-21T04:01:05.259 INFO:teuthology.orchestra.run.smithi133.stdout: mon pg warn max object skew = 0 2023-10-21T04:01:05.259 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.259 INFO:teuthology.orchestra.run.smithi133.stdout: # disable pg_autoscaler by default for new pools 2023-10-21T04:01:05.259 INFO:teuthology.orchestra.run.smithi133.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-21T04:01:05.260 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.260 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default size = 2 2023-10-21T04:01:05.260 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.260 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow primary affinity = true 2023-10-21T04:01:05.261 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow pg remap = true 2023-10-21T04:01:05.261 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on legacy crush tunables = false 2023-10-21T04:01:05.261 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on crush straw calc version zero = false 2023-10-21T04:01:05.261 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on no sortbitwise = false 2023-10-21T04:01:05.261 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on osd down out interval zero = false 2023-10-21T04:01:05.262 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on too few osds = false 2023-10-21T04:01:05.262 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-21T04:01:05.262 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-21T04:01:05.263 INFO:teuthology.orchestra.run.smithi133.stdout: mon_allow_pool_size_one = true 2023-10-21T04:01:05.263 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.263 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-21T04:01:05.263 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.264 INFO:teuthology.orchestra.run.smithi133.stdout: osd default data pool replay window = 5 2023-10-21T04:01:05.264 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.264 INFO:teuthology.orchestra.run.smithi133.stdout: mon allow pool delete = true 2023-10-21T04:01:05.264 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.264 INFO:teuthology.orchestra.run.smithi133.stdout: mon cluster log file level = debug 2023-10-21T04:01:05.265 INFO:teuthology.orchestra.run.smithi133.stdout: debug asserts on shutdown = true 2023-10-21T04:01:05.265 INFO:teuthology.orchestra.run.smithi133.stdout: mon health detail to clog = false 2023-10-21T04:01:05.265 INFO:teuthology.orchestra.run.smithi133.stdout: mon host = 172.21.15.133 2023-10-21T04:01:05.265 INFO:teuthology.orchestra.run.smithi133.stdout: mon client directed command retry = 5 2023-10-21T04:01:05.266 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on pool no app = False 2023-10-21T04:01:05.266 INFO:teuthology.orchestra.run.smithi133.stdout: ms inject socket failures = 5000 2023-10-21T04:01:05.266 INFO:teuthology.orchestra.run.smithi133.stdout: fsid = f3f3d351-07e7-41e0-8ea5-8c6ad08b8ea3 2023-10-21T04:01:05.266 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.267 INFO:teuthology.orchestra.run.smithi133.stdout:[osd] 2023-10-21T04:01:05.267 INFO:teuthology.orchestra.run.smithi133.stdout: osd journal size = 100 2023-10-21T04:01:05.267 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.268 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub load threshold = 5.0 2023-10-21T04:01:05.268 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub max interval = 600 2023-10-21T04:01:05.269 INFO:teuthology.orchestra.run.smithi133.stdout: osd mclock profile = high_recovery_ops 2023-10-21T04:01:05.269 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.269 INFO:teuthology.orchestra.run.smithi133.stdout: osd recover clone overlap = true 2023-10-21T04:01:05.270 INFO:teuthology.orchestra.run.smithi133.stdout: osd recovery max chunk = 1048576 2023-10-21T04:01:05.270 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.270 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug shutdown = true 2023-10-21T04:01:05.271 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug op order = true 2023-10-21T04:01:05.271 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify stray on activate = true 2023-10-21T04:01:05.271 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.271 INFO:teuthology.orchestra.run.smithi133.stdout: osd open classes on start = true 2023-10-21T04:01:05.272 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug pg log writeout = true 2023-10-21T04:01:05.272 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.272 INFO:teuthology.orchestra.run.smithi133.stdout: osd deep scrub update digest min age = 30 2023-10-21T04:01:05.272 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.273 INFO:teuthology.orchestra.run.smithi133.stdout: osd map max advance = 10 2023-10-21T04:01:05.273 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.273 INFO:teuthology.orchestra.run.smithi133.stdout: journal zero on create = true 2023-10-21T04:01:05.273 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.273 INFO:teuthology.orchestra.run.smithi133.stdout: filestore ondisk finisher threads = 3 2023-10-21T04:01:05.274 INFO:teuthology.orchestra.run.smithi133.stdout: filestore apply finisher threads = 3 2023-10-21T04:01:05.274 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.274 INFO:teuthology.orchestra.run.smithi133.stdout: bdev debug aio = true 2023-10-21T04:01:05.274 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug misdirected ops = true 2023-10-21T04:01:05.274 INFO:teuthology.orchestra.run.smithi133.stdout: bluestore block size = 96636764160 2023-10-21T04:01:05.275 INFO:teuthology.orchestra.run.smithi133.stdout: bluestore compression algorithm = lz4 2023-10-21T04:01:05.275 INFO:teuthology.orchestra.run.smithi133.stdout: bluestore compression mode = aggressive 2023-10-21T04:01:05.275 INFO:teuthology.orchestra.run.smithi133.stdout: bluestore fsck on mount = True 2023-10-21T04:01:05.275 INFO:teuthology.orchestra.run.smithi133.stdout: debug bluefs = 1/20 2023-10-21T04:01:05.275 INFO:teuthology.orchestra.run.smithi133.stdout: debug bluestore = 1/20 2023-10-21T04:01:05.276 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2023-10-21T04:01:05.276 INFO:teuthology.orchestra.run.smithi133.stdout: debug osd = 20 2023-10-21T04:01:05.276 INFO:teuthology.orchestra.run.smithi133.stdout: debug rocksdb = 4/10 2023-10-21T04:01:05.276 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd backfillfull_ratio = 0.85 2023-10-21T04:01:05.276 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd full ratio = 0.9 2023-10-21T04:01:05.276 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd nearfull ratio = 0.8 2023-10-21T04:01:05.277 INFO:teuthology.orchestra.run.smithi133.stdout: osd failsafe full ratio = 0.95 2023-10-21T04:01:05.277 INFO:teuthology.orchestra.run.smithi133.stdout: osd objectstore = bluestore 2023-10-21T04:01:05.277 INFO:teuthology.orchestra.run.smithi133.stdout: osd shutdown pgref assert = True 2023-10-21T04:01:05.277 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.277 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr] 2023-10-21T04:01:05.278 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2023-10-21T04:01:05.278 INFO:teuthology.orchestra.run.smithi133.stdout: debug mgr = 20 2023-10-21T04:01:05.278 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2023-10-21T04:01:05.279 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2023-10-21T04:01:05.279 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2023-10-21T04:01:05.279 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2023-10-21T04:01:05.279 INFO:teuthology.orchestra.run.smithi133.stdout: mgr/telemetry/nag = false 2023-10-21T04:01:05.279 INFO:teuthology.orchestra.run.smithi133.stdout: debug monc = 20 2023-10-21T04:01:05.280 INFO:teuthology.orchestra.run.smithi133.stdout: debug objecter = 20 2023-10-21T04:01:05.280 INFO:teuthology.orchestra.run.smithi133.stdout: debug rados = 20 2023-10-21T04:01:05.280 INFO:teuthology.orchestra.run.smithi133.stdout: debug rbd = 20 2023-10-21T04:01:05.280 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.281 INFO:teuthology.orchestra.run.smithi133.stdout:[mon] 2023-10-21T04:01:05.281 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2023-10-21T04:01:05.281 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2023-10-21T04:01:05.281 INFO:teuthology.orchestra.run.smithi133.stdout: debug paxos = 20 2023-10-21T04:01:05.281 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2023-10-21T04:01:05.282 INFO:teuthology.orchestra.run.smithi133.stdout: mon data avail warn = 5 2023-10-21T04:01:05.282 INFO:teuthology.orchestra.run.smithi133.stdout: mon mgr mkfs grace = 240 2023-10-21T04:01:05.282 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2023-10-21T04:01:05.282 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd reporter subtree level = osd 2023-10-21T04:01:05.282 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd prime pg temp = true 2023-10-21T04:01:05.283 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2023-10-21T04:01:05.283 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.283 INFO:teuthology.orchestra.run.smithi133.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-21T04:01:05.283 INFO:teuthology.orchestra.run.smithi133.stdout: auth mon ticket ttl = 660 # 11m 2023-10-21T04:01:05.283 INFO:teuthology.orchestra.run.smithi133.stdout: auth service ticket ttl = 240 # 4m 2023-10-21T04:01:05.283 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.284 INFO:teuthology.orchestra.run.smithi133.stdout: # don't complain about insecure global_id in the test suite 2023-10-21T04:01:05.284 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-21T04:01:05.284 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-21T04:01:05.284 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.284 INFO:teuthology.orchestra.run.smithi133.stdout: # 1m isn't quite enough 2023-10-21T04:01:05.285 INFO:teuthology.orchestra.run.smithi133.stdout: mon_down_mkfs_grace = 2m 2023-10-21T04:01:05.285 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.285 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_filestore_osds = false 2023-10-21T04:01:05.285 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:05.285 INFO:teuthology.orchestra.run.smithi133.stdout:[client] 2023-10-21T04:01:05.286 INFO:teuthology.orchestra.run.smithi133.stdout: rgw cache enabled = true 2023-10-21T04:01:05.286 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable ops log = true 2023-10-21T04:01:05.286 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable usage log = true 2023-10-21T04:01:05.286 INFO:teuthology.orchestra.run.smithi133.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-21T04:01:05.286 INFO:teuthology.orchestra.run.smithi133.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-21T04:01:05.286 INFO:teuthology.orchestra.run.smithi133.stdout: rbd default data pool = datapool 2023-10-21T04:01:05.287 INFO:teuthology.orchestra.run.smithi133.stdout: rbd default features = 1 2023-10-21T04:01:05.287 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.a] 2023-10-21T04:01:05.288 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-21T04:01:05.288 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-21T04:01:05.382 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-21T04:01:05.382 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:05.383 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-21T04:01:05.388 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:05.388 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-21T04:01:05.435 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi133.front.sepia.ceph.com 2023-10-21T04:01:05.436 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:05.436 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-21T04:01:05.436 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-21T04:01:05.498 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:05.498 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:01:05.546 INFO:tasks.ceph:Setting up mon nodes... 2023-10-21T04:01:05.547 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-21T04:01:05.547 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-21T04:01:05.630 INFO:teuthology.orchestra.run.smithi133.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-21T04:01:05.633 INFO:tasks.ceph:Setting up mds nodes... 2023-10-21T04:01:05.634 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-21T04:01:05.634 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-21T04:01:05.709 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-21T04:01:05.720 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-21T04:01:05.720 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-21T04:01:05.721 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-21T04:01:05.773 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T04:01:05.774 INFO:tasks.ceph:role: osd.0 2023-10-21T04:01:05.774 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com 2023-10-21T04:01:05.774 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-21T04:01:05.836 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-21T04:01:05.837 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-21T04:01:05.837 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-21T04:01:05.837 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-21T04:01:05.837 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-21T04:01:05.838 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2023-10-21T04:01:05.838 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-21T04:01:05.838 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-21T04:01:05.838 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-21T04:01:05.838 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-21T04:01:06.292 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2023-10-21T04:01:06.293 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2023-10-21T04:01:06.293 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-21T04:01:06.397 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-21T04:01:06.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:01:06.410 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: /sbin/restorecon: command not found 2023-10-21T04:01:06.410 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-21T04:01:06.465 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T04:01:06.466 INFO:tasks.ceph:role: osd.1 2023-10-21T04:01:06.466 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com 2023-10-21T04:01:06.466 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-21T04:01:06.530 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-21T04:01:06.530 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-21T04:01:06.530 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-21T04:01:06.530 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-21T04:01:06.531 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-21T04:01:06.531 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2023-10-21T04:01:06.531 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-21T04:01:06.531 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-21T04:01:06.531 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-21T04:01:06.532 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-21T04:01:06.997 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2023-10-21T04:01:06.999 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2023-10-21T04:01:07.000 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-21T04:01:07.136 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-21T04:01:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:01:07.148 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: /sbin/restorecon: command not found 2023-10-21T04:01:07.148 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-21T04:01:07.203 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-21T04:01:07.204 INFO:tasks.ceph:role: osd.2 2023-10-21T04:01:07.204 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi133.front.sepia.ceph.com 2023-10-21T04:01:07.204 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-21T04:01:07.265 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-21T04:01:07.265 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-21T04:01:07.266 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-21T04:01:07.266 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-21T04:01:07.266 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-21T04:01:07.266 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2023-10-21T04:01:07.266 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-21T04:01:07.267 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-21T04:01:07.267 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-21T04:01:07.267 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-21T04:01:07.735 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2023-10-21T04:01:07.737 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2023-10-21T04:01:07.737 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-21T04:01:07.858 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-21T04:01:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:01:07.870 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: /sbin/restorecon: command not found 2023-10-21T04:01:07.871 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:01:08.012 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:08.006+0000 7fabd741e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-21T04:01:08.013 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:08.006+0000 7fabd741e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-21T04:01:08.014 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:08.010+0000 7fabd741e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-21T04:01:08.014 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:08.010+0000 7fabd741e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-21T04:01:08.014 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:08.010+0000 7fabd741e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-21T04:01:08.014 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:08.010+0000 7fabd741e640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-21T04:01:11.026 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-21T04:01:11.040 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:01:11.120 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:11.114+0000 7fbe495c8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-21T04:01:11.120 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:11.114+0000 7fbe495c8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-21T04:01:11.121 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:11.114+0000 7fbe495c8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-21T04:01:11.121 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:11.114+0000 7fbe495c8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-21T04:01:11.121 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:11.114+0000 7fbe495c8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-21T04:01:11.121 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:11.114+0000 7fbe495c8640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-21T04:01:14.133 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-21T04:01:14.191 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:01:14.273 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:14.270+0000 7f56ddd07640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-21T04:01:14.274 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:14.270+0000 7f56ddd07640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-21T04:01:14.274 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:14.270+0000 7f56ddd07640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-21T04:01:14.275 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:14.270+0000 7f56ddd07640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-21T04:01:14.275 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:14.270+0000 7f56ddd07640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-21T04:01:14.275 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T04:01:14.270+0000 7f56ddd07640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-21T04:01:17.292 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-21T04:01:17.350 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-21T04:01:17.350 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:17.351 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-21T04:01:17.405 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:17.406 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-21T04:01:17.462 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:17.462 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-21T04:01:17.516 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:17.516 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-21T04:01:17.568 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:17.569 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-21T04:01:17.616 INFO:tasks.ceph:Adding keys to all mons... 2023-10-21T04:01:17.616 DEBUG:teuthology.orchestra.run.smithi133:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-21T04:01:17.668 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr.x] 2023-10-21T04:01:17.669 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQABTTNlOY2AJRAAobc6wglVDLdH0o6BA0yh1Q== 2023-10-21T04:01:17.669 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2023-10-21T04:01:17.669 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAETTNly7inABAAirixTu3ZlYvcT/o9D2uAkQ== 2023-10-21T04:01:17.669 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2023-10-21T04:01:17.669 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAHTTNlhMYLBxAAo09z/2X5E7PNOSC69lQIXg== 2023-10-21T04:01:17.670 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2023-10-21T04:01:17.670 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAKTTNlTXI4EBAAMRupOvc/DJC1cjKfJTduWQ== 2023-10-21T04:01:17.670 INFO:teuthology.orchestra.run.smithi133.stdout:[client.0] 2023-10-21T04:01:17.670 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQABTTNljQE2KhAAla1T/O5ZJVvkBgLOzoJjhw== 2023-10-21T04:01:17.671 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-21T04:01:17.757 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-21T04:01:17.842 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-21T04:01:17.925 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-21T04:01:18.011 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-21T04:01:18.099 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-21T04:01:18.099 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-21T04:01:18.111 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-21T04:01:18.202 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-21T04:01:18.255 DEBUG:teuthology.orchestra.run.smithi133:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-21T04:01:18.302 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-21T04:01:18.303 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-21T04:01:18.303 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-21T04:01:18.347 INFO:tasks.ceph.mon.a:Started 2023-10-21T04:01:18.347 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-21T04:01:18.347 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-21T04:01:18.348 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-21T04:01:18.350 INFO:tasks.ceph.mgr.x:Started 2023-10-21T04:01:18.351 INFO:tasks.ceph:Setting crush tunables to default 2023-10-21T04:01:18.351 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd crush tunables default 2023-10-21T04:01:18.606 INFO:teuthology.orchestra.run.smithi133.stderr:adjusted tunables profile to default 2023-10-21T04:01:18.629 INFO:tasks.ceph:check_enable_crimson: False 2023-10-21T04:01:18.630 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-21T04:01:18.630 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:18.630 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-21T04:01:18.643 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:18.643 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-21T04:01:18.697 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:18.697 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-21T04:01:18.731 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:18.726+0000 7fd25aed1180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-21T04:01:18.741 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new 096797d6-e2df-4521-8001-08e482bbe25c 0 2023-10-21T04:01:18.982 INFO:teuthology.orchestra.run.smithi133.stdout:0 2023-10-21T04:01:18.999 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new 7229425e-3c45-47b8-8e9e-d2c98d06a01f 1 2023-10-21T04:01:19.195 INFO:teuthology.orchestra.run.smithi133.stdout:1 2023-10-21T04:01:19.213 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new dfe1a0ba-0132-49bb-b531-389d265829eb 2 2023-10-21T04:01:19.278 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:19.274+0000 7fd25aed1180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-21T04:01:19.409 INFO:teuthology.orchestra.run.smithi133.stdout:2 2023-10-21T04:01:19.427 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-21T04:01:19.428 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-21T04:01:19.431 INFO:tasks.ceph.osd.0:Started 2023-10-21T04:01:19.432 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-21T04:01:19.432 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-21T04:01:19.434 INFO:tasks.ceph.osd.1:Started 2023-10-21T04:01:19.435 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-21T04:01:19.435 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-21T04:01:19.437 INFO:tasks.ceph.osd.2:Started 2023-10-21T04:01:19.438 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T04:01:19.466 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:19.462+0000 7fd25aed1180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-21T04:01:19.630 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:19.631 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":5,"fsid":"f3f3d351-07e7-41e0-8ea5-8c6ad08b8ea3","created":"2023-10-21T04:01:18.489015+0000","modified":"2023-10-21T04:01:19.406189+0000","last_up_change":"0.000000","last_in_change":"2023-10-21T04:01:19.406189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"096797d6-e2df-4521-8001-08e482bbe25c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7229425e-3c45-47b8-8e9e-d2c98d06a01f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dfe1a0ba-0132-49bb-b531-389d265829eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:01:19.647 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-21T04:01:19.647 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-21T04:01:19.949 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-21T04:01:20.038 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:20.034+0000 7fd25aed1180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-21T04:01:20.046 INFO:tasks.ceph.osd.2.smithi133.stderr:2023-10-21T04:01:20.042+0000 7f830328d640 -1 Falling back to public interface 2023-10-21T04:01:20.049 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:01:20.046+0000 7fc59f77a640 -1 Falling back to public interface 2023-10-21T04:01:20.070 INFO:tasks.ceph.osd.0.smithi133.stderr:2023-10-21T04:01:20.066+0000 7fea346b0640 -1 Falling back to public interface 2023-10-21T04:01:20.094 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:20.090+0000 7fd25aed1180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-21T04:01:20.133 INFO:teuthology.misc.health.smithi133.stdout: 2023-10-21T04:01:20.133 INFO:teuthology.misc.health.smithi133.stdout:{"epoch":5,"fsid":"f3f3d351-07e7-41e0-8ea5-8c6ad08b8ea3","created":"2023-10-21T04:01:18.489015+0000","modified":"2023-10-21T04:01:19.406189+0000","last_up_change":"0.000000","last_in_change":"2023-10-21T04:01:19.406189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"096797d6-e2df-4521-8001-08e482bbe25c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7229425e-3c45-47b8-8e9e-d2c98d06a01f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dfe1a0ba-0132-49bb-b531-389d265829eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:01:20.149 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-21T04:01:20.149 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:20.142+0000 7fd25aed1180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-21T04:01:20.225 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:20.218+0000 7fd25aed1180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-21T04:01:20.276 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:20.270+0000 7fd25aed1180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-21T04:01:20.352 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:20.346+0000 7fd25aed1180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-21T04:01:20.404 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:20.398+0000 7fd25aed1180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-21T04:01:20.542 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:20.538+0000 7fd25aed1180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-21T04:01:20.600 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:20.594+0000 7fd25aed1180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-21T04:01:21.242 INFO:tasks.ceph.mgr.x.smithi133.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-21T04:01:21.243 INFO:tasks.ceph.mgr.x.smithi133.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-21T04:01:21.243 INFO:tasks.ceph.mgr.x.smithi133.stderr: from numpy import show_config as show_numpy_config 2023-10-21T04:01:21.311 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:21.306+0000 7fd25aed1180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-21T04:01:21.480 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:21.474+0000 7fd25aed1180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-21T04:01:21.657 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:21.654+0000 7fd25aed1180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-21T04:01:21.726 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:21.722+0000 7fd25aed1180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-21T04:01:21.859 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:21.854+0000 7fd25aed1180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-21T04:01:22.478 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:22.474+0000 7fd25aed1180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-21T04:01:22.556 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:22.550+0000 7fd25aed1180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-21T04:01:22.742 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:22.738+0000 7fd25aed1180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-21T04:01:22.827 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:22.822+0000 7fd25aed1180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-21T04:01:23.145 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:23.138+0000 7fd25aed1180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-21T04:01:23.207 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:23.202+0000 7fd25aed1180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-21T04:01:23.489 INFO:tasks.ceph.osd.2.smithi133.stderr:2023-10-21T04:01:23.486+0000 7f830328d640 -1 osd.2 0 log_to_monitors true 2023-10-21T04:01:23.699 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:01:23.694+0000 7fc59f77a640 -1 osd.1 0 log_to_monitors true 2023-10-21T04:01:23.895 INFO:tasks.ceph.osd.0.smithi133.stderr:2023-10-21T04:01:23.890+0000 7fea346b0640 -1 osd.0 0 log_to_monitors true 2023-10-21T04:01:24.245 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:01:24.238+0000 7fd202742640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-21T04:01:26.452 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-21T04:01:26.570 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:01:26.566+0000 7fc59b722640 -1 osd.1 0 waiting for initial osdmap 2023-10-21T04:01:26.593 INFO:tasks.ceph.osd.2.smithi133.stderr:2023-10-21T04:01:26.586+0000 7f82ffa49640 -1 osd.2 0 waiting for initial osdmap 2023-10-21T04:01:26.605 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:01:26.602+0000 7fc593557640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-21T04:01:26.613 INFO:tasks.ceph.osd.2.smithi133.stderr:2023-10-21T04:01:26.610+0000 7f82f706a640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-21T04:01:26.638 INFO:tasks.ceph.osd.0.smithi133.stderr:2023-10-21T04:01:26.634+0000 7fea30658640 -1 osd.0 0 waiting for initial osdmap 2023-10-21T04:01:26.674 INFO:tasks.ceph.osd.0.smithi133.stderr:2023-10-21T04:01:26.666+0000 7fea2848d640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-21T04:01:26.759 INFO:teuthology.misc.health.smithi133.stdout: 2023-10-21T04:01:26.759 INFO:teuthology.misc.health.smithi133.stdout:{"epoch":7,"fsid":"f3f3d351-07e7-41e0-8ea5-8c6ad08b8ea3","created":"2023-10-21T04:01:18.489015+0000","modified":"2023-10-21T04:01:25.435153+0000","last_up_change":"0.000000","last_in_change":"2023-10-21T04:01:19.406189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"096797d6-e2df-4521-8001-08e482bbe25c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7229425e-3c45-47b8-8e9e-d2c98d06a01f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dfe1a0ba-0132-49bb-b531-389d265829eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:01:26.774 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-21T04:01:33.076 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-21T04:01:33.367 INFO:teuthology.misc.health.smithi133.stdout: 2023-10-21T04:01:33.368 INFO:teuthology.misc.health.smithi133.stdout:{"epoch":12,"fsid":"f3f3d351-07e7-41e0-8ea5-8c6ad08b8ea3","created":"2023-10-21T04:01:18.489015+0000","modified":"2023-10-21T04:01:31.607567+0000","last_up_change":"2023-10-21T04:01:27.594123+0000","last_in_change":"2023-10-21T04:01:19.406189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T04:01:29.307259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"096797d6-e2df-4521-8001-08e482bbe25c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6817","nonce":2130934728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6819","nonce":2130934728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6823","nonce":2130934728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6821","nonce":2130934728}]},"public_addr":"172.21.15.133:6817/2130934728","cluster_addr":"172.21.15.133:6819/2130934728","heartbeat_back_addr":"172.21.15.133:6823/2130934728","heartbeat_front_addr":"172.21.15.133:6821/2130934728","state":["exists","up"]},{"osd":1,"uuid":"7229425e-3c45-47b8-8e9e-d2c98d06a01f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6809","nonce":3846075335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6811","nonce":3846075335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6815","nonce":3846075335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6813","nonce":3846075335}]},"public_addr":"172.21.15.133:6809/3846075335","cluster_addr":"172.21.15.133:6811/3846075335","heartbeat_back_addr":"172.21.15.133:6815/3846075335","heartbeat_front_addr":"172.21.15.133:6813/3846075335","state":["exists","up"]},{"osd":2,"uuid":"dfe1a0ba-0132-49bb-b531-389d265829eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6801","nonce":350344564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6803","nonce":350344564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6807","nonce":350344564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6805","nonce":350344564}]},"public_addr":"172.21.15.133:6801/350344564","cluster_addr":"172.21.15.133:6803/350344564","heartbeat_back_addr":"172.21.15.133:6807/350344564","heartbeat_front_addr":"172.21.15.133:6805/350344564","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:01:24.866966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:01:24.660125+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:01:24.535867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:01:33.384 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-21T04:01:33.385 INFO:tasks.ceph:Creating RBD pool 2023-10-21T04:01:33.385 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-21T04:01:34.632 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'rbd' created 2023-10-21T04:01:34.658 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-21T04:01:35.640 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-21T04:01:35.670 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-21T04:01:35.670 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-21T04:01:35.671 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-21T04:01:35.671 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-21T04:01:35.672 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-21T04:01:36.002 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:36.019 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2374670033},{"type":"v1","addr":"172.21.15.133:6825","nonce":2374670033}]},"active_addr":"172.21.15.133:6825/2374670033","active_change":"2023-10-21T04:01:23.225117+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":3358300756}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":1086241106}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":4093318296}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":4232042389}]}]} 2023-10-21T04:01:36.021 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-21T04:01:36.022 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-21T04:01:36.022 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T04:01:36.283 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:36.283 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":14,"fsid":"f3f3d351-07e7-41e0-8ea5-8c6ad08b8ea3","created":"2023-10-21T04:01:18.489015+0000","modified":"2023-10-21T04:01:35.630697+0000","last_up_change":"2023-10-21T04:01:27.594123+0000","last_in_change":"2023-10-21T04:01:19.406189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T04:01:29.307259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-21T04:01:33.690298+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"096797d6-e2df-4521-8001-08e482bbe25c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6817","nonce":2130934728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6819","nonce":2130934728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6823","nonce":2130934728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6821","nonce":2130934728}]},"public_addr":"172.21.15.133:6817/2130934728","cluster_addr":"172.21.15.133:6819/2130934728","heartbeat_back_addr":"172.21.15.133:6823/2130934728","heartbeat_front_addr":"172.21.15.133:6821/2130934728","state":["exists","up"]},{"osd":1,"uuid":"7229425e-3c45-47b8-8e9e-d2c98d06a01f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6809","nonce":3846075335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6811","nonce":3846075335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6815","nonce":3846075335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6813","nonce":3846075335}]},"public_addr":"172.21.15.133:6809/3846075335","cluster_addr":"172.21.15.133:6811/3846075335","heartbeat_back_addr":"172.21.15.133:6815/3846075335","heartbeat_front_addr":"172.21.15.133:6813/3846075335","state":["exists","up"]},{"osd":2,"uuid":"dfe1a0ba-0132-49bb-b531-389d265829eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6801","nonce":350344564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6803","nonce":350344564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6807","nonce":350344564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6805","nonce":350344564}]},"public_addr":"172.21.15.133:6801/350344564","cluster_addr":"172.21.15.133:6803/350344564","heartbeat_back_addr":"172.21.15.133:6807/350344564","heartbeat_front_addr":"172.21.15.133:6805/350344564","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:01:24.866966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:01:24.660125+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:01:24.535867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:01:36.300 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-21T04:01:36.300 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T04:01:36.584 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:36.584 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":14,"fsid":"f3f3d351-07e7-41e0-8ea5-8c6ad08b8ea3","created":"2023-10-21T04:01:18.489015+0000","modified":"2023-10-21T04:01:35.630697+0000","last_up_change":"2023-10-21T04:01:27.594123+0000","last_in_change":"2023-10-21T04:01:19.406189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T04:01:29.307259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-21T04:01:33.690298+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"096797d6-e2df-4521-8001-08e482bbe25c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6817","nonce":2130934728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6819","nonce":2130934728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6823","nonce":2130934728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6821","nonce":2130934728}]},"public_addr":"172.21.15.133:6817/2130934728","cluster_addr":"172.21.15.133:6819/2130934728","heartbeat_back_addr":"172.21.15.133:6823/2130934728","heartbeat_front_addr":"172.21.15.133:6821/2130934728","state":["exists","up"]},{"osd":1,"uuid":"7229425e-3c45-47b8-8e9e-d2c98d06a01f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6809","nonce":3846075335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6811","nonce":3846075335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6815","nonce":3846075335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6813","nonce":3846075335}]},"public_addr":"172.21.15.133:6809/3846075335","cluster_addr":"172.21.15.133:6811/3846075335","heartbeat_back_addr":"172.21.15.133:6815/3846075335","heartbeat_front_addr":"172.21.15.133:6813/3846075335","state":["exists","up"]},{"osd":2,"uuid":"dfe1a0ba-0132-49bb-b531-389d265829eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6801","nonce":350344564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6803","nonce":350344564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6807","nonce":350344564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6805","nonce":350344564}]},"public_addr":"172.21.15.133:6801/350344564","cluster_addr":"172.21.15.133:6803/350344564","heartbeat_back_addr":"172.21.15.133:6807/350344564","heartbeat_front_addr":"172.21.15.133:6805/350344564","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:01:24.866966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:01:24.660125+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:01:24.535867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T04:01:36.603 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-21T04:01:36.604 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-21T04:01:36.604 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-21T04:01:36.753 INFO:teuthology.orchestra.run.smithi133.stdout:34359738373 2023-10-21T04:01:36.754 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-21T04:01:36.759 INFO:teuthology.orchestra.run.smithi133.stdout:34359738373 2023-10-21T04:01:36.759 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-21T04:01:36.767 INFO:teuthology.orchestra.run.smithi133.stdout:34359738372 2023-10-21T04:01:36.768 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-21T04:01:37.051 INFO:teuthology.orchestra.run.smithi133.stdout:34359738372 2023-10-21T04:01:37.068 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-10-21T04:01:37.083 INFO:teuthology.orchestra.run.smithi133.stdout:34359738371 2023-10-21T04:01:37.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2023-10-21T04:01:37.107 INFO:teuthology.orchestra.run.smithi133.stdout:34359738372 2023-10-21T04:01:37.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-10-21T04:01:38.069 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-21T04:01:38.101 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-21T04:01:38.125 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-21T04:01:38.363 INFO:teuthology.orchestra.run.smithi133.stdout:34359738373 2023-10-21T04:01:38.380 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-10-21T04:01:38.381 DEBUG:teuthology.parallel:result is None 2023-10-21T04:01:38.409 INFO:teuthology.orchestra.run.smithi133.stdout:34359738372 2023-10-21T04:01:38.426 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2023-10-21T04:01:38.426 DEBUG:teuthology.parallel:result is None 2023-10-21T04:01:38.441 INFO:teuthology.orchestra.run.smithi133.stdout:34359738373 2023-10-21T04:01:38.458 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-10-21T04:01:38.458 DEBUG:teuthology.parallel:result is None 2023-10-21T04:01:38.458 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-21T04:01:38.459 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T04:01:38.756 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:38.757 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T04:01:38.772 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-10-21T04:01:37.236022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81548,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033972,"statfs":{"total":289910292480,"available":289826787328,"internally_reserved":0,"allocated":483328,"data_stored":1281896,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.604904"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:01:35.659417+0000","last_change":"2023-10-21T04:01:35.655104+0000","last_active":"2023-10-21T04:01:35.659417+0000","last_peered":"2023-10-21T04:01:35.659417+0000","last_clean":"2023-10-21T04:01:35.659417+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T04:01:35.659417+0000","last_undegraded":"2023-10-21T04:01:35.659417+0000","last_fullsized":"2023-10-21T04:01:35.659417+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:47:03.130353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:01:35.659164+0000","last_change":"2023-10-21T04:01:35.652553+0000","last_active":"2023-10-21T04:01:35.659164+0000","last_peered":"2023-10-21T04:01:35.659164+0000","last_clean":"2023-10-21T04:01:35.659164+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T04:01:35.659164+0000","last_undegraded":"2023-10-21T04:01:35.659164+0000","last_fullsized":"2023-10-21T04:01:35.659164+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:08:56.378176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:01:35.658997+0000","last_change":"2023-10-21T04:01:35.651529+0000","last_active":"2023-10-21T04:01:35.658997+0000","last_peered":"2023-10-21T04:01:35.658997+0000","last_clean":"2023-10-21T04:01:35.658997+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T04:01:35.658997+0000","last_undegraded":"2023-10-21T04:01:35.658997+0000","last_fullsized":"2023-10-21T04:01:35.658997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:55:47.102256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:01:35.659123+0000","last_change":"2023-10-21T04:01:35.652367+0000","last_active":"2023-10-21T04:01:35.659123+0000","last_peered":"2023-10-21T04:01:35.659123+0000","last_clean":"2023-10-21T04:01:35.659123+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T04:01:35.659123+0000","last_undegraded":"2023-10-21T04:01:35.659123+0000","last_fullsized":"2023-10-21T04:01:35.659123+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:46:49.855234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:01:35.654226+0000","last_change":"2023-10-21T04:01:35.650136+0000","last_active":"2023-10-21T04:01:35.654226+0000","last_peered":"2023-10-21T04:01:35.654226+0000","last_clean":"2023-10-21T04:01:35.654226+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T04:01:35.654226+0000","last_undegraded":"2023-10-21T04:01:35.654226+0000","last_fullsized":"2023-10-21T04:01:35.654226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:58:09.196678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-21T04:01:36.644396+0000","last_change":"2023-10-21T04:01:35.652988+0000","last_active":"2023-10-21T04:01:36.644396+0000","last_peered":"2023-10-21T04:01:36.644396+0000","last_clean":"2023-10-21T04:01:36.644396+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T04:01:36.644396+0000","last_undegraded":"2023-10-21T04:01:36.644396+0000","last_fullsized":"2023-10-21T04:01:36.644396+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:39:12.668814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-21T04:01:36.644386+0000","last_change":"2023-10-21T04:01:35.650390+0000","last_active":"2023-10-21T04:01:36.644386+0000","last_peered":"2023-10-21T04:01:36.644386+0000","last_clean":"2023-10-21T04:01:36.644386+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T04:01:36.644386+0000","last_undegraded":"2023-10-21T04:01:36.644386+0000","last_fullsized":"2023-10-21T04:01:36.644386+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:08:09.428828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:01:35.653817+0000","last_change":"2023-10-21T04:01:35.650183+0000","last_active":"2023-10-21T04:01:35.653817+0000","last_peered":"2023-10-21T04:01:35.653817+0000","last_clean":"2023-10-21T04:01:35.653817+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T04:01:35.653817+0000","last_undegraded":"2023-10-21T04:01:35.653817+0000","last_fullsized":"2023-10-21T04:01:35.653817+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:52:20.538364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:01:34.633601+0000","last_change":"2023-10-21T04:01:30.998967+0000","last_active":"2023-10-21T04:01:34.633601+0000","last_peered":"2023-10-21T04:01:34.633601+0000","last_clean":"2023-10-21T04:01:34.633601+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T04:01:34.633601+0000","last_undegraded":"2023-10-21T04:01:34.633601+0000","last_fullsized":"2023-10-21T04:01:34.633601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:29.598046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:29.598046+0000","last_clean_scrub_stamp":"2023-10-21T04:01:29.598046+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:21:14.267398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T04:01:38.774 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T04:01:39.053 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:39.053 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T04:01:39.071 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-10-21T04:01:37.236022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81548,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033972,"statfs":{"total":289910292480,"available":289826787328,"internally_reserved":0,"allocated":483328,"data_stored":1281896,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.604904"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:01:35.659417+0000","last_change":"2023-10-21T04:01:35.655104+0000","last_active":"2023-10-21T04:01:35.659417+0000","last_peered":"2023-10-21T04:01:35.659417+0000","last_clean":"2023-10-21T04:01:35.659417+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T04:01:35.659417+0000","last_undegraded":"2023-10-21T04:01:35.659417+0000","last_fullsized":"2023-10-21T04:01:35.659417+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:47:03.130353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:01:35.659164+0000","last_change":"2023-10-21T04:01:35.652553+0000","last_active":"2023-10-21T04:01:35.659164+0000","last_peered":"2023-10-21T04:01:35.659164+0000","last_clean":"2023-10-21T04:01:35.659164+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T04:01:35.659164+0000","last_undegraded":"2023-10-21T04:01:35.659164+0000","last_fullsized":"2023-10-21T04:01:35.659164+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:08:56.378176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:01:35.658997+0000","last_change":"2023-10-21T04:01:35.651529+0000","last_active":"2023-10-21T04:01:35.658997+0000","last_peered":"2023-10-21T04:01:35.658997+0000","last_clean":"2023-10-21T04:01:35.658997+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T04:01:35.658997+0000","last_undegraded":"2023-10-21T04:01:35.658997+0000","last_fullsized":"2023-10-21T04:01:35.658997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:55:47.102256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:01:35.659123+0000","last_change":"2023-10-21T04:01:35.652367+0000","last_active":"2023-10-21T04:01:35.659123+0000","last_peered":"2023-10-21T04:01:35.659123+0000","last_clean":"2023-10-21T04:01:35.659123+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T04:01:35.659123+0000","last_undegraded":"2023-10-21T04:01:35.659123+0000","last_fullsized":"2023-10-21T04:01:35.659123+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:46:49.855234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:01:35.654226+0000","last_change":"2023-10-21T04:01:35.650136+0000","last_active":"2023-10-21T04:01:35.654226+0000","last_peered":"2023-10-21T04:01:35.654226+0000","last_clean":"2023-10-21T04:01:35.654226+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T04:01:35.654226+0000","last_undegraded":"2023-10-21T04:01:35.654226+0000","last_fullsized":"2023-10-21T04:01:35.654226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:58:09.196678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-21T04:01:36.644396+0000","last_change":"2023-10-21T04:01:35.652988+0000","last_active":"2023-10-21T04:01:36.644396+0000","last_peered":"2023-10-21T04:01:36.644396+0000","last_clean":"2023-10-21T04:01:36.644396+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T04:01:36.644396+0000","last_undegraded":"2023-10-21T04:01:36.644396+0000","last_fullsized":"2023-10-21T04:01:36.644396+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:39:12.668814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-21T04:01:36.644386+0000","last_change":"2023-10-21T04:01:35.650390+0000","last_active":"2023-10-21T04:01:36.644386+0000","last_peered":"2023-10-21T04:01:36.644386+0000","last_clean":"2023-10-21T04:01:36.644386+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T04:01:36.644386+0000","last_undegraded":"2023-10-21T04:01:36.644386+0000","last_fullsized":"2023-10-21T04:01:36.644386+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:08:09.428828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-21T04:01:35.653817+0000","last_change":"2023-10-21T04:01:35.650183+0000","last_active":"2023-10-21T04:01:35.653817+0000","last_peered":"2023-10-21T04:01:35.653817+0000","last_clean":"2023-10-21T04:01:35.653817+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T04:01:35.653817+0000","last_undegraded":"2023-10-21T04:01:35.653817+0000","last_fullsized":"2023-10-21T04:01:35.653817+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:52:20.538364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-21T04:01:34.633601+0000","last_change":"2023-10-21T04:01:30.998967+0000","last_active":"2023-10-21T04:01:34.633601+0000","last_peered":"2023-10-21T04:01:34.633601+0000","last_clean":"2023-10-21T04:01:34.633601+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T04:01:34.633601+0000","last_undegraded":"2023-10-21T04:01:34.633601+0000","last_fullsized":"2023-10-21T04:01:34.633601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:29.598046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:29.598046+0000","last_clean_scrub_stamp":"2023-10-21T04:01:29.598046+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:21:14.267398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T04:01:39.072 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-21T04:01:39.072 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-21T04:01:39.073 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-21T04:01:39.073 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-21T04:01:39.382 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T04:01:39.382 INFO:teuthology.orchestra.run.smithi133.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-21T04:01:39.397 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-21T04:01:39.398 INFO:teuthology.run_tasks:Running task exec... 2023-10-21T04:01:39.408 INFO:teuthology.task.exec:Executing custom commands... 2023-10-21T04:01:39.408 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi133.front.sepia.ceph.com 2023-10-21T04:01:39.409 DEBUG:teuthology.orchestra.run.smithi133:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4' 2023-10-21T04:01:40.674 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'datapool' created 2023-10-21T04:01:40.702 DEBUG:teuthology.orchestra.run.smithi133:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2023-10-21T04:01:43.719 INFO:teuthology.run_tasks:Running task install... 2023-10-21T04:01:43.728 DEBUG:teuthology.task.install:project ceph 2023-10-21T04:01:43.728 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}} 2023-10-21T04:01:43.728 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'} 2023-10-21T04:01:43.729 INFO:teuthology.task.install:Using flavor: default 2023-10-21T04:01:43.735 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-21T04:01:43.736 INFO:teuthology.task.install:extra packages: [] 2023-10-21T04:01:43.736 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-key list | grep Ceph 2023-10-21T04:01:43.802 INFO:teuthology.orchestra.run.smithi133.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-21T04:01:43.832 INFO:teuthology.orchestra.run.smithi133.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-21T04:01:43.832 INFO:teuthology.orchestra.run.smithi133.stdout:uid [ unknown] Ceph.com (release key) 2023-10-21T04:01:43.832 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-21T04:01:43.833 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2023-10-21T04:01:43.833 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:01:43.999 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default/ 2023-10-21T04:01:43.999 INFO:teuthology.task.install.deb:Package version is 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:01:44.179 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:01:44.180 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-21T04:01:44.194 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-get update 2023-10-21T04:01:44.477 INFO:teuthology.orchestra.run.smithi133.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy InRelease 2023-10-21T04:01:44.488 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-21T04:01:44.489 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-21T04:01:44.513 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:4 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release 2023-10-21T04:01:44.551 INFO:teuthology.orchestra.run.smithi133.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-21T04:01:44.561 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-21T04:01:44.643 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-21T04:01:45.696 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2023-10-21T04:01:45.710 DEBUG:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6744-g7f103c7c-1jammy cephadm=18.0.0-6744-g7f103c7c-1jammy ceph-mds=18.0.0-6744-g7f103c7c-1jammy ceph-mgr=18.0.0-6744-g7f103c7c-1jammy ceph-common=18.0.0-6744-g7f103c7c-1jammy ceph-fuse=18.0.0-6744-g7f103c7c-1jammy ceph-test=18.0.0-6744-g7f103c7c-1jammy ceph-volume=18.0.0-6744-g7f103c7c-1jammy radosgw=18.0.0-6744-g7f103c7c-1jammy python3-rados=18.0.0-6744-g7f103c7c-1jammy python3-rgw=18.0.0-6744-g7f103c7c-1jammy python3-cephfs=18.0.0-6744-g7f103c7c-1jammy python3-rbd=18.0.0-6744-g7f103c7c-1jammy libcephfs2=18.0.0-6744-g7f103c7c-1jammy libcephfs-dev=18.0.0-6744-g7f103c7c-1jammy librados2=18.0.0-6744-g7f103c7c-1jammy librbd1=18.0.0-6744-g7f103c7c-1jammy rbd-fuse=18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:01:45.773 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2023-10-21T04:01:45.955 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2023-10-21T04:01:45.956 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2023-10-21T04:01:46.113 INFO:teuthology.orchestra.run.smithi133.stdout:ceph is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.114 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-common is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.114 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-fuse is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.114 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-mds is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.114 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-mgr is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.115 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-test is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.115 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-volume is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.115 INFO:teuthology.orchestra.run.smithi133.stdout:cephadm is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.115 INFO:teuthology.orchestra.run.smithi133.stdout:libcephfs-dev is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.115 INFO:teuthology.orchestra.run.smithi133.stdout:libcephfs2 is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.116 INFO:teuthology.orchestra.run.smithi133.stdout:librados2 is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.116 INFO:teuthology.orchestra.run.smithi133.stdout:librbd1 is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.116 INFO:teuthology.orchestra.run.smithi133.stdout:python3-cephfs is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.116 INFO:teuthology.orchestra.run.smithi133.stdout:python3-rados is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.116 INFO:teuthology.orchestra.run.smithi133.stdout:python3-rbd is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.117 INFO:teuthology.orchestra.run.smithi133.stdout:python3-rgw is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.117 INFO:teuthology.orchestra.run.smithi133.stdout:radosgw is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.117 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-fuse is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-21T04:01:46.117 INFO:teuthology.orchestra.run.smithi133.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T04:01:46.117 INFO:teuthology.orchestra.run.smithi133.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-21T04:01:46.118 INFO:teuthology.orchestra.run.smithi133.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-21T04:01:46.118 INFO:teuthology.orchestra.run.smithi133.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-21T04:01:46.118 INFO:teuthology.orchestra.run.smithi133.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-21T04:01:46.118 INFO:teuthology.orchestra.run.smithi133.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-21T04:01:46.118 INFO:teuthology.orchestra.run.smithi133.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T04:01:46.150 INFO:teuthology.orchestra.run.smithi133.stdout:0 upgraded, 0 newly installed, 0 to remove and 257 not upgraded. 2023-10-21T04:01:46.151 INFO:teuthology.orchestra.run.smithi133.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-21T04:01:46.151 DEBUG:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2023-10-21T04:01:46.260 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2023-10-21T04:01:46.446 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2023-10-21T04:01:46.447 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2023-10-21T04:01:46.608 INFO:teuthology.orchestra.run.smithi133.stdout:The following packages were automatically installed and are no longer required: 2023-10-21T04:01:46.608 INFO:teuthology.orchestra.run.smithi133.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-10-21T04:01:46.609 INFO:teuthology.orchestra.run.smithi133.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-10-21T04:01:46.609 INFO:teuthology.orchestra.run.smithi133.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-10-21T04:01:46.610 INFO:teuthology.orchestra.run.smithi133.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-10-21T04:01:46.610 INFO:teuthology.orchestra.run.smithi133.stdout: python2.7 python2.7-dev python2.7-minimal 2023-10-21T04:01:46.610 INFO:teuthology.orchestra.run.smithi133.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-21T04:01:46.611 INFO:teuthology.orchestra.run.smithi133.stdout:The following additional packages will be installed: 2023-10-21T04:01:46.611 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemblk1 2023-10-21T04:01:46.611 INFO:teuthology.orchestra.run.smithi133.stdout:Suggested packages: 2023-10-21T04:01:46.612 INFO:teuthology.orchestra.run.smithi133.stdout: gnuplot gfio python-scipy 2023-10-21T04:01:46.638 INFO:teuthology.orchestra.run.smithi133.stdout:The following NEW packages will be installed: 2023-10-21T04:01:46.639 INFO:teuthology.orchestra.run.smithi133.stdout: fio libpmemblk1 2023-10-21T04:01:46.697 INFO:teuthology.orchestra.run.smithi133.stdout:0 upgraded, 2 newly installed, 0 to remove and 257 not upgraded. 2023-10-21T04:01:46.697 INFO:teuthology.orchestra.run.smithi133.stdout:Need to get 4,112 kB of archives. 2023-10-21T04:01:46.698 INFO:teuthology.orchestra.run.smithi133.stdout:After this operation, 7,813 kB of additional disk space will be used. 2023-10-21T04:01:46.698 INFO:teuthology.orchestra.run.smithi133.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-10-21T04:01:46.766 INFO:teuthology.orchestra.run.smithi133.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-10-21T04:01:47.270 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 4,112 kB in 0s (14.9 MB/s) 2023-10-21T04:01:47.429 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-10-21T04:01:47.533 INFO:teuthology.orchestra.run.smithi133.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172810 files and directories currently installed.) 2023-10-21T04:01:47.538 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-10-21T04:01:47.653 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-21T04:01:47.948 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package fio. 2023-10-21T04:01:47.975 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2023-10-21T04:01:48.006 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking fio (3.28-1) ... 2023-10-21T04:01:48.515 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-21T04:01:48.628 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up fio (3.28-1) ... 2023-10-21T04:01:49.190 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-21T04:01:50.223 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-21T04:01:50.671 INFO:teuthology.orchestra.run.smithi133.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-21T04:01:50.673 DEBUG:teuthology.parallel:result is None 2023-10-21T04:01:50.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:01:50.899 DEBUG:teuthology.orchestra.run.smithi133:> dpkg-query -W -f '${Version}' ceph 2023-10-21T04:01:50.918 INFO:teuthology.orchestra.run.smithi133.stdout:18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:01:50.918 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6744-g7f103c7c-1jammy 2023-10-21T04:01:50.919 INFO:teuthology.task.install:The correct ceph version 18.0.0-6744-g7f103c7c-1jammy is installed. 2023-10-21T04:01:50.919 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-21T04:01:50.920 INFO:teuthology.run_tasks:Running task workunit... 2023-10-21T04:01:50.939 INFO:tasks.workunit:Pulling workunits from ref 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:01:50.940 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-21T04:01:50.941 DEBUG:teuthology.orchestra.run.smithi133:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-21T04:01:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T04:01:50.968 INFO:teuthology.orchestra.run.smithi133.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-21T04:01:50.969 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-21T04:01:51.014 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-21T04:01:51.015 DEBUG:teuthology.orchestra.run.smithi133:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-21T04:01:51.062 INFO:tasks.workunit:timeout=3h 2023-10-21T04:01:51.063 INFO:tasks.workunit:cleanup=True 2023-10-21T04:01:51.063 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-21T04:01:51.112 INFO:tasks.workunit.client.0.smithi133.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-21T04:02:49.840 INFO:tasks.workunit.client.0.smithi133.stderr:Note: switching to '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'. 2023-10-21T04:02:49.841 INFO:tasks.workunit.client.0.smithi133.stderr: 2023-10-21T04:02:49.841 INFO:tasks.workunit.client.0.smithi133.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-21T04:02:49.841 INFO:tasks.workunit.client.0.smithi133.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-21T04:02:49.841 INFO:tasks.workunit.client.0.smithi133.stderr:state without impacting any branches by switching back to a branch. 2023-10-21T04:02:49.842 INFO:tasks.workunit.client.0.smithi133.stderr: 2023-10-21T04:02:49.842 INFO:tasks.workunit.client.0.smithi133.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-21T04:02:49.842 INFO:tasks.workunit.client.0.smithi133.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-21T04:02:49.842 INFO:tasks.workunit.client.0.smithi133.stderr: 2023-10-21T04:02:49.843 INFO:tasks.workunit.client.0.smithi133.stderr: git switch -c 2023-10-21T04:02:49.843 INFO:tasks.workunit.client.0.smithi133.stderr: 2023-10-21T04:02:49.843 INFO:tasks.workunit.client.0.smithi133.stderr:Or undo this operation with: 2023-10-21T04:02:49.843 INFO:tasks.workunit.client.0.smithi133.stderr: 2023-10-21T04:02:49.843 INFO:tasks.workunit.client.0.smithi133.stderr: git switch - 2023-10-21T04:02:49.843 INFO:tasks.workunit.client.0.smithi133.stderr: 2023-10-21T04:02:49.844 INFO:tasks.workunit.client.0.smithi133.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-21T04:02:49.844 INFO:tasks.workunit.client.0.smithi133.stderr: 2023-10-21T04:02:49.844 INFO:tasks.workunit.client.0.smithi133.stderr:HEAD is now at 7f103c7cad2 qa/suites/rbd/cli/workloads/rbd_support_module_recovery: more debug 2023-10-21T04:02:49.845 DEBUG:teuthology.orchestra.run.smithi133:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-21T04:02:49.897 INFO:tasks.workunit.client.0.smithi133.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-21T04:02:49.899 INFO:tasks.workunit.client.0.smithi133.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-21T04:02:49.899 INFO:tasks.workunit.client.0.smithi133.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-21T04:02:50.690 INFO:tasks.workunit.client.0.smithi133.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-21T04:02:50.770 INFO:tasks.workunit.client.0.smithi133.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-21T04:02:50.821 INFO:tasks.workunit.client.0.smithi133.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-21T04:02:50.822 INFO:tasks.workunit.client.0.smithi133.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-21T04:02:50.822 INFO:tasks.workunit.client.0.smithi133.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-21T04:02:50.867 INFO:tasks.workunit.client.0.smithi133.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-21T04:02:50.870 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-21T04:02:50.870 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-21T04:02:50.918 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-21T04:02:50.919 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-21T04:02:50.920 DEBUG:teuthology.orchestra.run.smithi133:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-21T04:02:50.971 INFO:tasks.workunit.client.0.smithi133.stderr:+ POOL=rbd 2023-10-21T04:02:50.971 INFO:tasks.workunit.client.0.smithi133.stderr:+ IMAGE_PREFIX=image 2023-10-21T04:02:50.971 INFO:tasks.workunit.client.0.smithi133.stderr:+ NUM_IMAGES=20 2023-10-21T04:02:50.972 INFO:tasks.workunit.client.0.smithi133.stderr:+ RUN_TIME=3600 2023-10-21T04:02:50.972 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror pool enable rbd image 2023-10-21T04:02:51.019 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-21T04:02:51.054 INFO:tasks.workunit.client.0.smithi133.stdout:2e49ab08-f50e-4193-90d2-ce85c3f79091 2023-10-21T04:02:51.057 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i = 1 )) 2023-10-21T04:02:51.057 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:02:51.058 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image1 2023-10-21T04:02:51.105 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-21T04:02:51.548 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:02:51.561 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-21T04:02:51.624 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:02:51.624 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:02:51.625 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image2 2023-10-21T04:02:51.673 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-21T04:02:52.554 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:02:52.568 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-21T04:02:52.629 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:02:52.630 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:02:52.630 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image3 2023-10-21T04:02:52.677 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-21T04:02:53.565 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:02:53.577 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-21T04:02:53.638 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:02:53.638 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:02:53.638 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image4 2023-10-21T04:02:53.685 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-21T04:02:54.569 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:02:54.582 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-21T04:02:54.644 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:02:54.644 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:02:54.644 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image5 2023-10-21T04:02:54.693 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-21T04:02:55.580 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:02:55.593 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-21T04:02:55.653 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:02:55.653 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:02:55.654 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image6 2023-10-21T04:02:55.705 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-21T04:02:56.579 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:02:56.592 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-21T04:02:56.653 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:02:56.653 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:02:56.654 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image7 2023-10-21T04:02:56.702 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-21T04:02:57.590 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:02:57.605 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-21T04:02:57.665 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:02:57.665 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:02:57.665 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image8 2023-10-21T04:02:57.713 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-21T04:02:58.595 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:02:58.609 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-21T04:02:58.669 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:02:58.669 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:02:58.669 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image9 2023-10-21T04:02:58.716 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-21T04:02:59.613 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:02:59.626 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-21T04:02:59.686 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:02:59.687 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:02:59.687 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image10 2023-10-21T04:02:59.737 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-21T04:03:00.616 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:03:00.631 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-21T04:03:00.692 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:00.692 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:00.693 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image11 2023-10-21T04:03:00.741 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-21T04:03:01.630 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:03:01.646 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-21T04:03:01.707 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:01.708 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:01.708 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image12 2023-10-21T04:03:01.756 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-21T04:03:02.635 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:03:02.649 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-21T04:03:02.711 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:02.711 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:02.712 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image13 2023-10-21T04:03:02.759 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-21T04:03:03.645 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:03:03.660 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-21T04:03:03.721 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:03.721 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:03.722 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image14 2023-10-21T04:03:03.771 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-21T04:03:04.646 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:03:04.659 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-21T04:03:04.723 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:04.723 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:04.723 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image15 2023-10-21T04:03:04.771 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-21T04:03:05.659 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:03:05.674 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-21T04:03:05.737 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:05.737 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:05.737 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image16 2023-10-21T04:03:05.786 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-21T04:03:06.662 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:03:06.675 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-21T04:03:06.737 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:06.737 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:06.737 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image17 2023-10-21T04:03:06.785 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-21T04:03:07.668 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:03:07.682 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-21T04:03:07.745 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:07.745 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:07.745 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image18 2023-10-21T04:03:07.793 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-21T04:03:08.673 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:03:08.686 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-21T04:03:08.751 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:08.751 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:08.751 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image19 2023-10-21T04:03:08.802 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-21T04:03:09.696 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:03:09.707 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-21T04:03:09.769 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:09.769 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:09.769 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image20 2023-10-21T04:03:09.817 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-21T04:03:10.697 INFO:tasks.workunit.client.0.smithi133.stdout:Mirroring enabled 2023-10-21T04:03:10.708 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-21T04:03:10.770 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:10.770 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:10.771 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i = 1 )) 2023-10-21T04:03:10.771 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:10.771 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image1 2023-10-21T04:03:10.908 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-21T04:03:10.908 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:10.909 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:10.909 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:10.909 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image2 2023-10-21T04:03:10.982 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-21T04:03:10.983 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:10.983 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:10.983 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:10.983 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image3 2023-10-21T04:03:11.054 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-21T04:03:11.055 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:11.055 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:11.055 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:11.055 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image4 2023-10-21T04:03:11.129 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-21T04:03:11.129 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:11.129 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:11.129 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:11.130 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image5 2023-10-21T04:03:11.201 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-21T04:03:11.201 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:11.202 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:11.202 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:11.202 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image6 2023-10-21T04:03:11.285 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-21T04:03:11.285 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:11.285 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:11.286 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:11.286 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image7 2023-10-21T04:03:11.407 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-21T04:03:11.407 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:11.408 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:11.408 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image8 2023-10-21T04:03:11.408 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:11.509 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-21T04:03:11.509 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:11.509 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:11.509 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:11.510 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image9 2023-10-21T04:03:11.619 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-21T04:03:11.619 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:11.619 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:11.620 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image10 2023-10-21T04:03:11.622 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:11.733 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-21T04:03:11.733 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:11.733 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:11.733 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:11.734 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image11 2023-10-21T04:03:11.849 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-21T04:03:11.849 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:11.849 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:11.850 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:11.850 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image12 2023-10-21T04:03:11.990 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-21T04:03:11.991 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:11.991 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:11.991 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:11.992 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image13 2023-10-21T04:03:12.108 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-21T04:03:12.109 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:12.109 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:12.110 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:12.110 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image14 2023-10-21T04:03:12.259 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-21T04:03:12.259 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:12.259 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:12.259 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image15 2023-10-21T04:03:12.260 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:12.405 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-21T04:03:12.406 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:12.406 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:12.406 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image16 2023-10-21T04:03:12.406 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:12.534 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-21T04:03:12.535 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:12.535 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:12.535 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image17 2023-10-21T04:03:12.545 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:12.679 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-21T04:03:12.679 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:12.679 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:12.680 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:12.680 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image18 2023-10-21T04:03:12.828 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-21T04:03:12.828 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:12.828 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:12.838 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image19 2023-10-21T04:03:12.839 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:12.987 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-21T04:03:12.987 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:12.988 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:12.988 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:12.994 INFO:tasks.workunit.client.0.smithi133.stderr:++ sudo rbd device map rbd/image20 2023-10-21T04:03:13.174 INFO:tasks.workunit.client.0.smithi133.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-21T04:03:13.175 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T04:03:13.175 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 20 )) 2023-10-21T04:03:13.175 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:03:13.176 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697860993 2023-10-21T04:03:13.176 INFO:tasks.workunit.client.0.smithi133.stderr:+ END_TIME=1697864593 2023-10-21T04:03:13.176 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR= 2023-10-21T04:03:13.177 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR= 2023-10-21T04:03:13.177 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:03:13.177 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n '' ]] 2023-10-21T04:03:13.177 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:03:13.178 INFO:tasks.workunit.client.0.smithi133.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-21T04:03:23.190 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:03:23.191 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:03:23.194 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:03:23.217 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:03:24.064 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/4093318296 2023-10-21T04:03:24.064 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:03:24.065 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861004 2023-10-21T04:03:24.065 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:03:24.065 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/4093318296 ]] 2023-10-21T04:03:24.065 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/4093318296 != '' ]] 2023-10-21T04:03:24.066 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/4093318296 2023-10-21T04:03:26.493 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/4093318296 until 2023-10-21T05:03:25.631736+0000 (3600 sec) 2023-10-21T04:03:26.537 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:03:26.541 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/4093318296 2023-10-21T04:03:27.290 INFO:tasks.workunit.client.0.smithi133.stderr:listed 1 entries 2023-10-21T04:03:27.326 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/4093318296 2023-10-21T04:03:27.327 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:03:37.328 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:03:37.328 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:03:37.342 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:03:37.344 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:03:38.195 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/4093318296 2023-10-21T04:03:38.195 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:03:38.196 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861018 2023-10-21T04:03:38.196 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:03:38.196 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/4093318296 ]] 2023-10-21T04:03:38.196 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/4093318296 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\4\0\9\3\3\1\8\2\9\6 ]] 2023-10-21T04:03:38.197 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:03:48.198 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:03:48.203 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:03:48.204 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:03:48.211 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:03:49.132 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/4093318296 2023-10-21T04:03:49.133 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:03:49.133 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861029 2023-10-21T04:03:49.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:03:49.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/4093318296 ]] 2023-10-21T04:03:49.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/4093318296 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\4\0\9\3\3\1\8\2\9\6 ]] 2023-10-21T04:03:49.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:03:59.137 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:03:59.138 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:03:59.146 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:03:59.147 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:04:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.004+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.004+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562262f5c580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.018 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.012+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.019 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.012+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562262f5cf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.026 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.016+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.026 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.016+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226284a100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.030 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.020+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.030 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.020+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.036 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.028+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.036 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.028+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.042 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.032+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.042 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.032+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.048 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.040+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.048 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.040+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.055 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.048+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.055 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.048+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.061 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.052+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.062 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.052+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.070 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/4093318296 2023-10-21T04:04:00.071 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:04:00.071 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861040 2023-10-21T04:04:00.072 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:04:00.072 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/4093318296 ]] 2023-10-21T04:04:00.072 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/4093318296 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\4\0\9\3\3\1\8\2\9\6 ]] 2023-10-21T04:04:00.072 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:04:00.073 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.064+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.073 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.064+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226284a280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.074 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.064+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.074 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.064+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562262f5cd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.080 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.072+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.080 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.072+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562262f5cf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.085 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.076+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.085 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.076+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.090 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.084+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.090 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.084+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.097 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.088+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.098 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.088+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.101 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.092+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.101 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.092+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.107 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.100+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.107 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.100+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.114 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.104+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.114 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.104+0000 7fd1f4e27640 -1 librbd::ImageState: 0x5622626faa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.123 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.116+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.124 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.116+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.127 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.120+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:00.127 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:04:00.120+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:04:10.073 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:04:10.074 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:04:10.076 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:04:10.077 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:04:10.949 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/4093318296 2023-10-21T04:04:10.949 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:04:10.950 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861050 2023-10-21T04:04:10.950 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:04:10.951 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/4093318296 ]] 2023-10-21T04:04:10.951 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/4093318296 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\4\0\9\3\3\1\8\2\9\6 ]] 2023-10-21T04:04:10.952 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:04:20.952 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:04:20.960 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:04:20.971 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:04:20.975 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:04:21.896 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/4093318296 2023-10-21T04:04:21.896 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:04:21.896 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861061 2023-10-21T04:04:21.897 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:04:21.897 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/4093318296 ]] 2023-10-21T04:04:21.897 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/4093318296 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\4\0\9\3\3\1\8\2\9\6 ]] 2023-10-21T04:04:21.897 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:04:31.899 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:04:31.900 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:04:31.906 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:04:31.933 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:04:32.710 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/154413538 2023-10-21T04:04:32.710 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:04:32.711 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861072 2023-10-21T04:04:32.711 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:04:32.711 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/154413538 ]] 2023-10-21T04:04:32.711 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/154413538 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\4\0\9\3\3\1\8\2\9\6 ]] 2023-10-21T04:04:32.712 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/154413538 2023-10-21T04:04:34.545 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/154413538 until 2023-10-21T05:04:33.736636+0000 (3600 sec) 2023-10-21T04:04:34.583 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:04:34.591 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/154413538 2023-10-21T04:04:35.330 INFO:tasks.workunit.client.0.smithi133.stderr:listed 2 entries 2023-10-21T04:04:35.360 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/154413538 2023-10-21T04:04:35.360 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:04:45.362 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:04:45.363 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:04:45.366 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:04:45.386 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:04:46.297 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/154413538 2023-10-21T04:04:46.298 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:04:46.300 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861086 2023-10-21T04:04:46.300 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:04:46.300 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/154413538 ]] 2023-10-21T04:04:46.300 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/154413538 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\5\4\4\1\3\5\3\8 ]] 2023-10-21T04:04:46.301 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:04:55.309 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:04:55.300+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:04:55.309 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:04:55.300+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:04:55.309 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:04:55.300+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:04:55.310 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:04:55.300+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:04:55.310 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:04:55.300+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:04:55.310 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:04:55.300+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:04:55.310 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:04:55.300+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:04:55.310 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:04:55.300+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:04:56.302 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:04:56.310 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:04:56.317 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:04:56.338 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:04:57.139 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/154413538 2023-10-21T04:04:57.139 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:04:57.139 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861097 2023-10-21T04:04:57.140 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:04:57.140 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/154413538 ]] 2023-10-21T04:04:57.140 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/154413538 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\5\4\4\1\3\5\3\8 ]] 2023-10-21T04:04:57.140 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:05:00.011 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.004+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.004+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622631cbc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.004+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.004+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263431580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.022 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.012+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.022 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.012+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263371980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.031 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.024+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.032 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.024+0000 7fd1f4626640 -1 librbd::ImageState: 0x56226284a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.048 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.040+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.048 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.040+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263371d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.057 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.048+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.058 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.048+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263370480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.062 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.052+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.062 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.052+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263431380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.067 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.060+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.067 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.060+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263370180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.070 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.060+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.070 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.060+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263371e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.073 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.064+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.074 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.064+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263370800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.080 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.072+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.081 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.072+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263371980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.089 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.080+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.090 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.080+0000 7fd1f4626640 -1 librbd::ImageState: 0x56226284a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.095 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.088+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.096 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.088+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263431480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.105 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.096+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.105 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.096+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263370480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.116 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.108+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.116 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.108+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263371600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.121 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.112+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.122 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.112+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263371880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.132 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.124+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.132 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.124+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263371c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.141 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.132+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.141 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.132+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263371c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.149 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.140+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.149 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.140+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263431880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.155 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.148+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:00.156 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:00.148+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263431680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:05:07.146 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:05:07.146 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:05:07.146 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:05:07.149 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:05:08.132 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/154413538 2023-10-21T04:05:08.132 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:05:08.133 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861108 2023-10-21T04:05:08.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:05:08.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/154413538 ]] 2023-10-21T04:05:08.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/154413538 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\5\4\4\1\3\5\3\8 ]] 2023-10-21T04:05:08.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:05:18.136 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:05:18.136 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:05:18.136 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:05:18.155 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:05:18.972 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/154413538 2023-10-21T04:05:18.972 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:05:18.978 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861118 2023-10-21T04:05:18.978 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:05:18.978 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/154413538 ]] 2023-10-21T04:05:18.979 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/154413538 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\5\4\4\1\3\5\3\8 ]] 2023-10-21T04:05:18.979 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:05:28.980 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:05:28.980 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:05:28.981 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:05:28.982 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:05:29.869 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/154413538 2023-10-21T04:05:29.870 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:05:29.870 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861129 2023-10-21T04:05:29.871 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:05:29.871 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/154413538 ]] 2023-10-21T04:05:29.871 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/154413538 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\5\4\4\1\3\5\3\8 ]] 2023-10-21T04:05:29.871 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:05:39.886 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:05:39.886 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:05:39.886 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:05:39.899 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:05:40.959 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3000627311 2023-10-21T04:05:40.960 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:05:40.961 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861140 2023-10-21T04:05:40.961 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:05:40.962 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3000627311 ]] 2023-10-21T04:05:40.962 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3000627311 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\5\4\4\1\3\5\3\8 ]] 2023-10-21T04:05:40.962 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/3000627311 2023-10-21T04:05:42.528 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/3000627311 until 2023-10-21T05:05:41.810082+0000 (3600 sec) 2023-10-21T04:05:42.566 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/3000627311 2023-10-21T04:05:42.568 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:05:43.421 INFO:tasks.workunit.client.0.smithi133.stderr:listed 3 entries 2023-10-21T04:05:43.456 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/3000627311 2023-10-21T04:05:43.456 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:05:53.458 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:05:53.461 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:05:53.464 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:05:53.467 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:05:54.403 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3000627311 2023-10-21T04:05:54.404 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:05:54.404 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861154 2023-10-21T04:05:54.404 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:05:54.405 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3000627311 ]] 2023-10-21T04:05:54.405 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3000627311 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\0\0\6\2\7\3\1\1 ]] 2023-10-21T04:05:54.405 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:06:00.010 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:59.999+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.011 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:05:59.999+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350a100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.003+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.015 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.003+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.023 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.011+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.024 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.011+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350a500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.037 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.027+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.037 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.027+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350a700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.043 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.031+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.043 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.031+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350a900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.058 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.047+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.059 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.047+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263205080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.070 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.059+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.070 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.059+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263204b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.076 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.067+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.077 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.067+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263204f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.094 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.083+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.095 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.083+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350a580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.105 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.095+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.105 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.095+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.120 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.111+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.120 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.111+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350a100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.126 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.115+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.126 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.115+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263370700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.130 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.119+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.130 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.119+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350a700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.145 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.135+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.145 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.135+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350ab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.158 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.147+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.159 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.147+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350ad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.167 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.155+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.168 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.155+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350af80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.175 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.163+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.175 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.163+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350b180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.184 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.175+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.184 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.175+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350b380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.197 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.187+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.197 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.187+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350bd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.201 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.191+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:00.202 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:06:00.191+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226350bc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:06:04.411 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:06:04.412 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:06:04.417 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:06:04.431 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:06:05.292 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3000627311 2023-10-21T04:06:05.294 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:06:05.299 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861165 2023-10-21T04:06:05.300 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:06:05.300 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3000627311 ]] 2023-10-21T04:06:05.300 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3000627311 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\0\0\6\2\7\3\1\1 ]] 2023-10-21T04:06:05.300 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:06:09.679 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:06:09.679 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:06:09.680 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:06:09.680 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:06:09.680 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:06:09.680 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:06:09.680 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:06:09.681 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:06:09.681 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:06:09.681 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:06:09.681 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:06:09.681 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:06:09.682 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:06:09.682 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:09.667+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:06:15.304 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:06:15.305 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:06:15.305 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:06:15.305 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:06:16.186 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3000627311 2023-10-21T04:06:16.187 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:06:16.187 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861176 2023-10-21T04:06:16.187 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:06:16.188 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3000627311 ]] 2023-10-21T04:06:16.188 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3000627311 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\0\0\6\2\7\3\1\1 ]] 2023-10-21T04:06:16.188 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:06:19.288 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:06:19.289 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:06:19.289 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:06:19.289 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:06:19.289 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:06:19.290 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:06:19.290 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:06:19.290 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:06:19.290 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:06:19.290 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:06:19.291 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:06:19.291 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:06:19.293 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:06:19.293 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:06:19.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:06:26.194 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:06:26.200 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:06:26.201 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:06:26.207 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:06:27.092 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3000627311 2023-10-21T04:06:27.093 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:06:27.093 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861187 2023-10-21T04:06:27.093 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:06:27.094 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3000627311 ]] 2023-10-21T04:06:27.094 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3000627311 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\0\0\6\2\7\3\1\1 ]] 2023-10-21T04:06:27.094 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:06:37.095 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:06:37.097 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:06:37.103 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:06:37.111 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:06:38.036 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3000627311 2023-10-21T04:06:38.036 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:06:38.037 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861198 2023-10-21T04:06:38.037 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:06:38.038 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3000627311 ]] 2023-10-21T04:06:38.038 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3000627311 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\0\0\6\2\7\3\1\1 ]] 2023-10-21T04:06:38.038 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:06:48.044 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:06:48.044 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:06:48.044 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:06:48.050 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:06:48.989 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3000627311 2023-10-21T04:06:48.989 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:06:48.990 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861208 2023-10-21T04:06:48.990 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:06:48.990 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3000627311 ]] 2023-10-21T04:06:48.990 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3000627311 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\0\0\6\2\7\3\1\1 ]] 2023-10-21T04:06:48.991 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:06:58.996 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:06:58.996 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:06:58.997 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:06:59.000 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:06:59.424 INFO:tasks.workunit.client.0.smithi133.stderr:2023-10-21T04:06:59.419+0000 7f52477fe640 0 --2- 172.21.15.133:0/3079398359 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5248124600 0x7f5248126810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-21T04:07:00.157 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3937471728 2023-10-21T04:07:00.157 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:07:00.158 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861220 2023-10-21T04:07:00.158 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:00.159 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3937471728 ]] 2023-10-21T04:07:00.159 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3937471728 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\0\0\6\2\7\3\1\1 ]] 2023-10-21T04:07:00.159 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/3937471728 2023-10-21T04:07:01.611 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.603+0000 7fd1f4626640 -1 librbd::ImageWatcher: 0x5622631ce000 image watch failed: 94705694574592, (107) Transport endpoint is not connected 2023-10-21T04:07:01.611 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.603+0000 7fd1f4626640 -1 librbd::Watcher: 0x5622631ce000 handle_error: handle=94705694574592: (107) Transport endpoint is not connected 2023-10-21T04:07:01.611 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.603+0000 7fd1f4626640 -1 librbd::ImageWatcher: 0x5622631cef00 image watch failed: 94705689729024, (107) Transport endpoint is not connected 2023-10-21T04:07:01.611 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.603+0000 7fd1f4626640 -1 librbd::Watcher: 0x5622631cef00 handle_error: handle=94705689729024: (107) Transport endpoint is not connected 2023-10-21T04:07:01.612 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.603+0000 7fd1f4626640 -1 librbd::watcher::RewatchRequest: 0x5622633f6910 handle_unwatch client blocklisted 2023-10-21T04:07:01.612 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.603+0000 7fd1f4626640 -1 librbd::Watcher: 0x5622631ce000 handle_rewatch: client blocklisted 2023-10-21T04:07:01.612 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.603+0000 7fd1f4626640 -1 librbd::watcher::RewatchRequest: 0x56226344bb80 handle_unwatch client blocklisted 2023-10-21T04:07:01.612 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.603+0000 7fd1f4626640 -1 librbd::Watcher: 0x5622631cef00 handle_rewatch: client blocklisted 2023-10-21T04:07:01.613 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.607+0000 7fd1f4626640 -1 librbd::ImageWatcher: 0x5622632c2600 image watch failed: 94705696986112, (107) Transport endpoint is not connected 2023-10-21T04:07:01.613 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.607+0000 7fd1f4626640 -1 librbd::Watcher: 0x5622632c2600 handle_error: handle=94705696986112: (107) Transport endpoint is not connected 2023-10-21T04:07:01.613 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.607+0000 7fd1f4626640 -1 librbd::ImageWatcher: 0x5622632c2900 image watch failed: 94705696984064, (107) Transport endpoint is not connected 2023-10-21T04:07:01.614 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.607+0000 7fd1f4626640 -1 librbd::Watcher: 0x5622632c2900 handle_error: handle=94705696984064: (107) Transport endpoint is not connected 2023-10-21T04:07:01.614 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.607+0000 7fd1f4626640 -1 librbd::watcher::RewatchRequest: 0x56226322f950 handle_unwatch client blocklisted 2023-10-21T04:07:01.614 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.607+0000 7fd1f4626640 -1 librbd::Watcher: 0x5622632c2600 handle_rewatch: client blocklisted 2023-10-21T04:07:01.614 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.607+0000 7fd1f4626640 -1 librbd::watcher::RewatchRequest: 0x562262e734f0 handle_unwatch client blocklisted 2023-10-21T04:07:01.614 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.607+0000 7fd1f4626640 -1 librbd::Watcher: 0x5622632c2900 handle_rewatch: client blocklisted 2023-10-21T04:07:01.620 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.615+0000 7fd1f3e25640 -1 librbd::ImageWatcher: 0x5622632c3200 image watch failed: 94705699462144, (107) Transport endpoint is not connected 2023-10-21T04:07:01.620 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.615+0000 7fd1f3e25640 -1 librbd::Watcher: 0x5622632c3200 handle_error: handle=94705699462144: (107) Transport endpoint is not connected 2023-10-21T04:07:01.621 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.615+0000 7fd1f3e25640 -1 librbd::ImageWatcher: 0x5622632c2f00 image watch failed: 94705689723904, (107) Transport endpoint is not connected 2023-10-21T04:07:01.621 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.615+0000 7fd1f3e25640 -1 librbd::Watcher: 0x5622632c2f00 handle_error: handle=94705689723904: (107) Transport endpoint is not connected 2023-10-21T04:07:01.621 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.615+0000 7fd1f3e25640 -1 librbd::ImageWatcher: 0x5622632c2000 image watch failed: 94705693243392, (107) Transport endpoint is not connected 2023-10-21T04:07:01.621 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.615+0000 7fd1f3e25640 -1 librbd::Watcher: 0x5622632c2000 handle_error: handle=94705693243392: (107) Transport endpoint is not connected 2023-10-21T04:07:01.622 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.615+0000 7fd1f3e25640 -1 librbd::watcher::RewatchRequest: 0x562262905bd0 handle_unwatch client blocklisted 2023-10-21T04:07:01.622 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.615+0000 7fd1f3e25640 -1 librbd::Watcher: 0x5622632c3200 handle_rewatch: client blocklisted 2023-10-21T04:07:01.622 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.615+0000 7fd1f3e25640 -1 librbd::watcher::RewatchRequest: 0x562262905720 handle_unwatch client blocklisted 2023-10-21T04:07:01.623 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.615+0000 7fd1f3e25640 -1 librbd::Watcher: 0x5622632c2f00 handle_rewatch: client blocklisted 2023-10-21T04:07:01.623 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.615+0000 7fd1f3e25640 -1 librbd::watcher::RewatchRequest: 0x562262905b80 handle_unwatch client blocklisted 2023-10-21T04:07:01.623 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.615+0000 7fd1f3e25640 -1 librbd::Watcher: 0x5622632c2000 handle_rewatch: client blocklisted 2023-10-21T04:07:01.625 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.619+0000 7fd1f3e25640 -1 librbd::ImageWatcher: 0x562262bb8300 image watch failed: 94705699521536, (107) Transport endpoint is not connected 2023-10-21T04:07:01.625 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.619+0000 7fd1f3e25640 -1 librbd::Watcher: 0x562262bb8300 handle_error: handle=94705699521536: (107) Transport endpoint is not connected 2023-10-21T04:07:01.626 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.619+0000 7fd1f3e25640 -1 librbd::ImageWatcher: 0x562262bb8000 image watch failed: 94705696980992, (107) Transport endpoint is not connected 2023-10-21T04:07:01.626 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.619+0000 7fd1f3e25640 -1 librbd::Watcher: 0x562262bb8000 handle_error: handle=94705696980992: (107) Transport endpoint is not connected 2023-10-21T04:07:01.626 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.619+0000 7fd1f3e25640 -1 librbd::watcher::RewatchRequest: 0x5622633ec320 handle_unwatch client blocklisted 2023-10-21T04:07:01.627 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.619+0000 7fd1f3e25640 -1 librbd::Watcher: 0x562262bb8300 handle_rewatch: client blocklisted 2023-10-21T04:07:01.627 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.619+0000 7fd1f3e25640 -1 librbd::watcher::RewatchRequest: 0x56226311da40 handle_unwatch client blocklisted 2023-10-21T04:07:01.627 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.619+0000 7fd1f3e25640 -1 librbd::Watcher: 0x562262bb8000 handle_rewatch: client blocklisted 2023-10-21T04:07:01.628 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.623+0000 7fd1f4626640 -1 librbd::ImageWatcher: 0x5622632c2300 image watch failed: 94705693237248, (107) Transport endpoint is not connected 2023-10-21T04:07:01.628 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.623+0000 7fd1f4626640 -1 librbd::Watcher: 0x5622632c2300 handle_error: handle=94705693237248: (107) Transport endpoint is not connected 2023-10-21T04:07:01.629 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.623+0000 7fd1f4626640 -1 librbd::watcher::RewatchRequest: 0x562262905c20 handle_unwatch client blocklisted 2023-10-21T04:07:01.629 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:01.623+0000 7fd1f4626640 -1 librbd::Watcher: 0x5622632c2300 handle_rewatch: client blocklisted 2023-10-21T04:07:02.603 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/3937471728 until 2023-10-21T05:07:01.787097+0000 (3600 sec) 2023-10-21T04:07:02.614 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.614 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bc270 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.615 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x5622637cc160 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.615 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.615 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bc9c0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.615 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x5622639838c0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.615 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.616 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bc8f0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.616 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x562263898f20 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.616 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.616 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.616 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.617 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bc000 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.617 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x562263899e40 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.617 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bc820 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.617 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x5622638994a0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.617 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bc1a0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.618 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.607+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x562263898160 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.619 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.619 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.619 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.620 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.620 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.620 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.620 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x562263983760 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.621 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x562263398a80 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.621 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x562263899080 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.621 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x562263983080 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.621 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x562263982160 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.622 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x56226363b4a0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.622 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x5622637cc160 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.622 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x562263980b00 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.622 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5622633043c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.622 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x562262b31740 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.623 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x562263980a80 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.623 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.611+0000 7fd1f4626640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x56226392a340 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.630 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.630 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.630 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.631 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bc750 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.631 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::io::AioCompletion: 0x5622637cc840 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.631 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bc4e0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.631 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::io::AioCompletion: 0x5622635cd340 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.631 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bca90 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.632 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::io::AioCompletion: 0x562263983ce0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.632 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::managed_lock::ReleaseRequest: 0x562262fdd800 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.632 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::managed_lock::ReleaseRequest: 0x562263464d80 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.632 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::managed_lock::ReleaseRequest: 0x562262b69200 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.632 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::managed_lock::ReleaseRequest: 0x56226346e660 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.633 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::managed_lock::ReleaseRequest: 0x56226346f1a0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.633 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.615+0000 7fd1f3e25640 -1 librbd::managed_lock::ReleaseRequest: 0x562262b69c80 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.633 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.623+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.633 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.623+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263674e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.634 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.623+0000 7fd1f4626640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.634 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.623+0000 7fd1f4626640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bc340 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.635 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.623+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x562263983b80 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.635 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.623+0000 7fd1f4626640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.635 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.623+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x562263899080 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.635 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.623+0000 7fd1f3e25640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x562262bb6400 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.635 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.623+0000 7fd1f3e25640 -1 librbd::managed_lock::ReleaseRequest: 0x562262b64060 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.636 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:07:02.636 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/3937471728 2023-10-21T04:07:02.636 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.631+0000 7fd1f3e25640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.637 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.631+0000 7fd1f3e25640 -1 librbd::io::AioCompletion: 0x562263983e40 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.637 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.631+0000 7fd1f3e25640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5622629978c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.637 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.631+0000 7fd1f3e25640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.638 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.631+0000 7fd1f3e25640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.638 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.631+0000 7fd1f3e25640 -1 librbd::io::AioCompletion: 0x5622639838c0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.638 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.631+0000 7fd1f3e25640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x562262bb6380 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.638 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.631+0000 7fd1f3e25640 -1 librbd::io::AioCompletion: 0x5622637cc160 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.639 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.631+0000 7fd1f3e25640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x562262bb6340 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.643 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.635+0000 7fd1f3e25640 -1 librbd::managed_lock::ReleaseRequest: 0x562262f8fc80 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.643 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.635+0000 7fd1f3e25640 -1 librbd::managed_lock::ReleaseRequest: 0x562262e13ec0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:02.644 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:07:02.635+0000 7fd1f3e25640 -1 librbd::managed_lock::ReleaseRequest: 0x562263115b60 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:07:03.258 INFO:tasks.workunit.client.0.smithi133.stderr:listed 4 entries 2023-10-21T04:07:03.283 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/3937471728 2023-10-21T04:07:03.284 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:07:13.299 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:13.301 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:13.302 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:13.303 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:07:14.253 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3937471728 2023-10-21T04:07:14.253 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:07:14.255 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861234 2023-10-21T04:07:14.255 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:14.256 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3937471728 ]] 2023-10-21T04:07:14.256 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3937471728 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\9\3\7\4\7\1\7\2\8 ]] 2023-10-21T04:07:14.256 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:07:24.261 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:24.263 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:24.265 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:24.285 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:07:25.148 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3937471728 2023-10-21T04:07:25.148 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:07:25.149 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861245 2023-10-21T04:07:25.149 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:25.149 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3937471728 ]] 2023-10-21T04:07:25.149 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3937471728 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\9\3\7\4\7\1\7\2\8 ]] 2023-10-21T04:07:25.150 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:07:35.151 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:35.152 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:35.155 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:07:35.160 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:36.131 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3937471728 2023-10-21T04:07:36.132 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:07:36.133 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861256 2023-10-21T04:07:36.133 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:36.133 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3937471728 ]] 2023-10-21T04:07:36.133 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3937471728 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\9\3\7\4\7\1\7\2\8 ]] 2023-10-21T04:07:36.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:07:46.139 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:07:46.150 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:46.151 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:46.158 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:47.088 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3937471728 2023-10-21T04:07:47.088 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:07:47.089 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861267 2023-10-21T04:07:47.089 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:47.089 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3937471728 ]] 2023-10-21T04:07:47.089 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3937471728 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\9\3\7\4\7\1\7\2\8 ]] 2023-10-21T04:07:47.089 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:07:57.095 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:07:57.096 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:07:57.096 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:07:57.100 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:07:57.999 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3937471728 2023-10-21T04:07:58.000 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:07:58.000 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861277 2023-10-21T04:07:58.000 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:07:58.000 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3937471728 ]] 2023-10-21T04:07:58.001 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3937471728 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\9\3\7\4\7\1\7\2\8 ]] 2023-10-21T04:07:58.001 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:08:08.002 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:08.004 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:08:08.006 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:08.018 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:08.941 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/374453278 2023-10-21T04:08:08.942 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:08:08.942 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861288 2023-10-21T04:08:08.943 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:08.943 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/374453278 ]] 2023-10-21T04:08:08.943 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/374453278 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\9\3\7\4\7\1\7\2\8 ]] 2023-10-21T04:08:08.943 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/374453278 2023-10-21T04:08:10.579 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/374453278 until 2023-10-21T05:08:09.800079+0000 (3600 sec) 2023-10-21T04:08:10.627 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:08:10.627 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/374453278 2023-10-21T04:08:11.450 INFO:tasks.workunit.client.0.smithi133.stderr:listed 5 entries 2023-10-21T04:08:11.481 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/374453278 2023-10-21T04:08:11.481 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:08:21.483 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:08:21.483 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:21.487 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:21.502 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:22.372 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/374453278 2023-10-21T04:08:22.374 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:08:22.374 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861302 2023-10-21T04:08:22.375 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:22.375 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/374453278 ]] 2023-10-21T04:08:22.375 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/374453278 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\7\4\4\5\3\2\7\8 ]] 2023-10-21T04:08:22.375 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:08:29.880 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:08:29.880 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:08:29.880 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:08:29.880 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:08:29.881 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:08:29.881 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:08:29.881 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:08:29.881 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:08:29.881 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:08:29.882 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:08:29.882 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:08:29.882 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:08:29.882 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:08:29.882 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:29.866+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:08:32.381 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:08:32.381 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:32.381 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:32.390 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:33.252 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/374453278 2023-10-21T04:08:33.253 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:08:33.253 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861313 2023-10-21T04:08:33.253 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:33.254 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/374453278 ]] 2023-10-21T04:08:33.254 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/374453278 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\7\4\4\5\3\2\7\8 ]] 2023-10-21T04:08:33.254 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:08:43.256 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:08:43.256 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:43.260 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:43.262 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:44.239 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/374453278 2023-10-21T04:08:44.239 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:08:44.240 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861324 2023-10-21T04:08:44.241 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:44.241 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/374453278 ]] 2023-10-21T04:08:44.241 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/374453278 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\7\4\4\5\3\2\7\8 ]] 2023-10-21T04:08:44.241 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:08:53.834 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:08:53.826+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:08:54.248 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:08:54.249 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:08:54.253 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:08:54.254 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:08:55.072 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/374453278 2023-10-21T04:08:55.073 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:08:55.073 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861335 2023-10-21T04:08:55.074 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:08:55.074 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/374453278 ]] 2023-10-21T04:08:55.074 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/374453278 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\7\4\4\5\3\2\7\8 ]] 2023-10-21T04:08:55.074 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:09:00.010 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.002+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.011 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.002+0000 7fd1f4626640 -1 librbd::ImageState: 0x56226346ca00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.017 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.010+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.017 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.010+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263494900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.024 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.014+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.024 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.014+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263675180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.031 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.022+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.032 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.022+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263675e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.039 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.030+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.039 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.030+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263675d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.048 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.038+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.048 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.038+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263675b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.054 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.046+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.055 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.046+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.061 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.054+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.061 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.054+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368a380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.070 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.058+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.070 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.058+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368a580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.074 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.066+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.074 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.066+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368ac80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.079 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.070+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.079 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.070+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368ab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.083 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.074+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.083 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.074+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368a900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.087 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.078+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.087 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.078+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368ad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.091 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.082+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.092 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.082+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368af80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.103 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.094+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.103 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.094+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368b180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.109 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.098+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.109 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.098+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368b380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.113 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.106+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.113 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.106+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368b580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.118 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.110+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.118 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.110+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368b780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.122 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.114+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.123 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.114+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368b980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.127 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.118+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:00.127 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:09:00.118+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368bb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:09:05.078 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:09:05.078 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:05.079 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:05.080 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:06.015 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/374453278 2023-10-21T04:09:06.015 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:09:06.016 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861346 2023-10-21T04:09:06.016 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:06.016 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/374453278 ]] 2023-10-21T04:09:06.016 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/374453278 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\7\4\4\5\3\2\7\8 ]] 2023-10-21T04:09:06.016 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:09:16.019 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:09:16.020 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:16.022 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:16.024 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:16.884 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/210740177 2023-10-21T04:09:16.884 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:09:16.885 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861356 2023-10-21T04:09:16.885 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:16.885 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/210740177 ]] 2023-10-21T04:09:16.886 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/210740177 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\7\4\4\5\3\2\7\8 ]] 2023-10-21T04:09:16.886 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/210740177 2023-10-21T04:09:18.582 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/210740177 until 2023-10-21T05:09:17.770792+0000 (3600 sec) 2023-10-21T04:09:18.621 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:09:18.629 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/210740177 2023-10-21T04:09:19.399 INFO:tasks.workunit.client.0.smithi133.stderr:listed 6 entries 2023-10-21T04:09:19.424 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/210740177 2023-10-21T04:09:19.424 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:09:29.432 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:09:29.433 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:29.436 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:29.437 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:30.301 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/210740177 2023-10-21T04:09:30.301 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:09:30.302 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861370 2023-10-21T04:09:30.302 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:30.302 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/210740177 ]] 2023-10-21T04:09:30.302 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/210740177 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\1\0\7\4\0\1\7\7 ]] 2023-10-21T04:09:30.302 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:09:40.303 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:09:40.304 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:40.304 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:40.337 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:41.213 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/210740177 2023-10-21T04:09:41.213 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:09:41.224 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861381 2023-10-21T04:09:41.224 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:41.224 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/210740177 ]] 2023-10-21T04:09:41.224 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/210740177 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\1\0\7\4\0\1\7\7 ]] 2023-10-21T04:09:41.224 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:09:51.226 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:09:51.227 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:09:51.233 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:09:51.241 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:09:52.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/210740177 2023-10-21T04:09:52.135 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:09:52.135 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861392 2023-10-21T04:09:52.135 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:09:52.136 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/210740177 ]] 2023-10-21T04:09:52.136 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/210740177 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\1\0\7\4\0\1\7\7 ]] 2023-10-21T04:09:52.136 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:10:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.005+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.015 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.005+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622638c6080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.017 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.005+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.017 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.005+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622638c6280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.021 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.013+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.021 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.013+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622638c6480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.026 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.017+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.026 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.017+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c6680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.030 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.021+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.030 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.021+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c6880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.034 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.025+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.034 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.025+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c6a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.043 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.029+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.043 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.029+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c7080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.043 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.033+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.044 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.033+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c6f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.047 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.037+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.047 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.037+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c7600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.052 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.041+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.052 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.041+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c7480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.057 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.049+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.057 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.049+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c7280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.061 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.053+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.061 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.053+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c6d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.066 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.057+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.066 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.057+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c7880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.070 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.061+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.071 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.061+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c7a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.083 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.073+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.084 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.073+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c7c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.089 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.081+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.089 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.081+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622638c7e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.100 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.089+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.100 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.089+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263896080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.105 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.097+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.105 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.097+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263896280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.116 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.105+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.117 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.105+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263896480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.123 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.113+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:00.124 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:10:00.113+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263896680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:10:02.137 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:10:02.137 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:02.137 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:02.143 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:03.091 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/210740177 2023-10-21T04:10:03.091 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:10:03.092 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861403 2023-10-21T04:10:03.092 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:03.092 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/210740177 ]] 2023-10-21T04:10:03.092 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/210740177 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\1\0\7\4\0\1\7\7 ]] 2023-10-21T04:10:03.093 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:10:13.094 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:13.095 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:13.099 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:10:13.105 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:14.010 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/210740177 2023-10-21T04:10:14.011 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:10:14.012 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861414 2023-10-21T04:10:14.012 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:14.013 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/210740177 ]] 2023-10-21T04:10:14.013 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/210740177 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\1\0\7\4\0\1\7\7 ]] 2023-10-21T04:10:14.013 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:10:24.026 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:10:24.026 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:24.026 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:24.027 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:24.951 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1058636164 2023-10-21T04:10:24.952 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:10:24.960 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861424 2023-10-21T04:10:24.961 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:24.961 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1058636164 ]] 2023-10-21T04:10:24.961 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1058636164 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\1\0\7\4\0\1\7\7 ]] 2023-10-21T04:10:24.961 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/1058636164 2023-10-21T04:10:26.596 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/1058636164 until 2023-10-21T05:10:25.792155+0000 (3600 sec) 2023-10-21T04:10:26.632 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:10:26.632 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/1058636164 2023-10-21T04:10:27.427 INFO:tasks.workunit.client.0.smithi133.stderr:listed 7 entries 2023-10-21T04:10:27.466 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/1058636164 2023-10-21T04:10:27.466 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:10:37.480 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:37.483 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:37.486 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:37.490 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:10:38.394 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1058636164 2023-10-21T04:10:38.395 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:10:38.396 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861438 2023-10-21T04:10:38.396 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:38.396 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1058636164 ]] 2023-10-21T04:10:38.396 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1058636164 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\5\8\6\3\6\1\6\4 ]] 2023-10-21T04:10:38.396 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:10:40.988 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:40.977+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:10:40.988 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:40.977+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:10:48.399 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:10:48.399 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:48.399 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:10:48.400 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:49.416 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1058636164 2023-10-21T04:10:49.417 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:10:49.417 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861449 2023-10-21T04:10:49.418 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:10:49.418 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1058636164 ]] 2023-10-21T04:10:49.418 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1058636164 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\5\8\6\3\6\1\6\4 ]] 2023-10-21T04:10:49.418 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:10:55.518 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:10:55.520 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:10:55.520 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:10:55.520 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:10:55.520 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:10:55.521 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:10:55.521 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:10:55.521 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:10:55.521 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:10:55.521 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:10:55.522 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:10:55.522 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:10:55.522 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:55.505+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:10:57.851 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:57.841+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:10:57.852 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:57.841+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:10:57.852 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:57.841+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:10:59.214 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:10:59.214 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:10:59.217 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:10:59.217 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:10:59.217 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:10:59.218 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:10:59.218 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:10:59.218 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:10:59.218 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:10:59.218 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:10:59.219 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:10:59.219 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:10:59.219 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:10:59.219 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:10:59.205+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:10:59.419 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:10:59.420 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:10:59.421 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:10:59.443 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:00.015 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.005+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.016 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.005+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.016 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.005+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263494f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.016 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.005+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263929b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.017 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.005+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.017 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.005+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368be80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.018 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.009+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.019 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.009+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263928d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.022 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.013+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.023 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.013+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368aa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.024 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.013+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.024 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.013+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368bf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.029 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.021+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.029 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.021+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263675880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.041 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.029+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.041 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.029+0000 7fd1f4e27640 -1 librbd::ImageState: 0x56226368ae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.052 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.041+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.053 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.041+0000 7fd1f4e27640 -1 librbd::ImageState: 0x5622635cab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.056 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.045+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.056 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.045+0000 7fd1f4e27640 -1 librbd::ImageState: 0x5622635cb380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.062 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.053+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.063 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.053+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263675c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.067 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.057+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.067 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.057+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263674c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.074 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.065+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.074 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.065+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622638c7c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.079 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.069+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.079 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.069+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622638c7e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.085 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.073+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.085 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.073+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622635cb900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.091 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.081+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.092 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.081+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263896300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.097 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.089+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.098 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.089+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263896000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.104 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.093+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.104 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.093+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263896980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.112 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.101+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.112 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.101+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263897380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.117 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.105+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.118 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:11:00.105+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263897200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:11:00.294 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1058636164 2023-10-21T04:11:00.295 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:11:00.295 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861460 2023-10-21T04:11:00.296 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:00.296 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1058636164 ]] 2023-10-21T04:11:00.296 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1058636164 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\5\8\6\3\6\1\6\4 ]] 2023-10-21T04:11:00.296 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:11:03.991 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:11:03.981+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:11:03.995 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:11:03.985+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:11:10.305 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:10.305 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:10.306 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:10.309 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:11:11.226 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1058636164 2023-10-21T04:11:11.226 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:11:11.227 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861471 2023-10-21T04:11:11.227 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:11.227 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1058636164 ]] 2023-10-21T04:11:11.227 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1058636164 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\5\8\6\3\6\1\6\4 ]] 2023-10-21T04:11:11.228 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:11:21.229 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:21.232 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:21.233 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:11:21.241 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:22.224 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1058636164 2023-10-21T04:11:22.224 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:11:22.225 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861482 2023-10-21T04:11:22.225 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:22.225 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1058636164 ]] 2023-10-21T04:11:22.225 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1058636164 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\5\8\6\3\6\1\6\4 ]] 2023-10-21T04:11:22.225 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:11:32.228 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:11:32.229 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:32.230 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:32.237 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:33.127 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1058636164 2023-10-21T04:11:33.128 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:11:33.128 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861493 2023-10-21T04:11:33.128 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:33.129 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1058636164 ]] 2023-10-21T04:11:33.129 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1058636164 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\5\8\6\3\6\1\6\4 ]] 2023-10-21T04:11:33.129 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:11:43.129 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:11:43.131 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:43.135 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:43.147 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:44.046 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/4278077090 2023-10-21T04:11:44.047 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:11:44.047 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861504 2023-10-21T04:11:44.047 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:44.048 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/4278077090 ]] 2023-10-21T04:11:44.048 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/4278077090 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\5\8\6\3\6\1\6\4 ]] 2023-10-21T04:11:44.048 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/4278077090 2023-10-21T04:11:46.616 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/4278077090 until 2023-10-21T05:11:45.923824+0000 (3600 sec) 2023-10-21T04:11:46.652 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:11:46.652 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/4278077090 2023-10-21T04:11:47.523 INFO:tasks.workunit.client.0.smithi133.stderr:listed 8 entries 2023-10-21T04:11:47.561 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/4278077090 2023-10-21T04:11:47.562 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:11:57.563 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:11:57.569 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:11:57.570 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:11:57.590 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:11:58.589 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/4278077090 2023-10-21T04:11:58.589 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:11:58.589 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861518 2023-10-21T04:11:58.590 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:11:58.590 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/4278077090 ]] 2023-10-21T04:11:58.590 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/4278077090 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\4\2\7\8\0\7\7\0\9\0 ]] 2023-10-21T04:11:58.590 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:12:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.005+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.005+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.005+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263494e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.015 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.005+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263929f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.021 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.009+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.021 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.009+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263929e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.031 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.021+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.032 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.021+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263928600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.042 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.033+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.043 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.033+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622635caf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.049 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.041+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.050 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.041+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562262f5dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.062 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.053+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.063 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.053+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263446400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.073 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.065+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.074 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.065+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263205600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.080 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.069+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.080 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.069+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263896900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.085 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.073+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.085 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.073+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263897b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.095 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.085+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.095 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.085+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263897d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.101 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.089+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.101 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.089+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263446380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.110 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.101+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.110 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.101+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263897f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.117 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.105+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.117 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.105+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263576180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.122 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.113+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.122 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.113+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263576380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.125 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.113+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.125 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.113+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263576580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.135 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.125+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.136 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.125+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263576780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.141 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.133+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.142 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.133+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263576980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.151 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.141+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.151 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.141+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263576b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.154 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.145+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:00.155 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:12:00.145+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263576d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:12:08.607 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:08.609 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:12:08.611 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:08.612 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:09.589 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/4278077090 2023-10-21T04:12:09.589 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:12:09.590 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861529 2023-10-21T04:12:09.590 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:09.590 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/4278077090 ]] 2023-10-21T04:12:09.590 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/4278077090 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\4\2\7\8\0\7\7\0\9\0 ]] 2023-10-21T04:12:09.591 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:12:15.948 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:12:15.949 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:12:15.949 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:12:15.949 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:12:15.949 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:12:15.950 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:12:15.950 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:12:15.950 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:12:15.950 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:12:15.950 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:12:15.951 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:12:15.951 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:12:15.951 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:12:15.951 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:15.937+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:12:19.594 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:12:19.598 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:19.604 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:19.609 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:20.583 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/4278077090 2023-10-21T04:12:20.583 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:12:20.584 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861540 2023-10-21T04:12:20.584 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:20.584 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/4278077090 ]] 2023-10-21T04:12:20.585 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/4278077090 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\4\2\7\8\0\7\7\0\9\0 ]] 2023-10-21T04:12:20.585 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:12:30.586 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:30.589 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:30.590 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:12:30.601 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:31.476 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/4278077090 2023-10-21T04:12:31.477 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:12:31.478 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861551 2023-10-21T04:12:31.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:31.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/4278077090 ]] 2023-10-21T04:12:31.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/4278077090 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\4\2\7\8\0\7\7\0\9\0 ]] 2023-10-21T04:12:31.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:12:32.611 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:12:32.601+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:12:41.486 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:12:41.496 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:41.497 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:41.503 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:42.421 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/102508650 2023-10-21T04:12:42.422 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:12:42.423 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861562 2023-10-21T04:12:42.423 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:42.423 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/102508650 ]] 2023-10-21T04:12:42.423 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/102508650 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\4\2\7\8\0\7\7\0\9\0 ]] 2023-10-21T04:12:42.424 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/102508650 2023-10-21T04:12:44.677 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/102508650 until 2023-10-21T05:12:43.906446+0000 (3600 sec) 2023-10-21T04:12:44.706 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/102508650 2023-10-21T04:12:44.713 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:12:45.623 INFO:tasks.workunit.client.0.smithi133.stderr:listed 9 entries 2023-10-21T04:12:45.672 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/102508650 2023-10-21T04:12:45.672 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:12:55.677 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:12:55.678 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:12:55.681 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:12:55.682 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:12:56.694 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/102508650 2023-10-21T04:12:56.695 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:12:56.695 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861576 2023-10-21T04:12:56.696 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:12:56.696 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/102508650 ]] 2023-10-21T04:12:56.696 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/102508650 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\2\5\0\8\6\5\0 ]] 2023-10-21T04:12:56.697 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:13:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.005+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.005+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263446680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.016 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.005+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.017 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.005+0000 7fd1f4626640 -1 librbd::ImageState: 0x56226368ab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.022 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.013+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.023 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.013+0000 7fd1f4626640 -1 librbd::ImageState: 0x56226368ab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.027 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.017+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.027 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.017+0000 7fd1f4626640 -1 librbd::ImageState: 0x56226368b900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.034 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.025+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.034 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.025+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263896c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.041 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.033+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.041 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.033+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263896d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.047 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.037+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.048 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.037+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263495f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.051 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.041+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.051 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.041+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263896500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.065 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.057+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.066 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.057+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.066 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.057+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622638c7480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.066 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.057+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622638c7b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.088 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.077+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.088 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.077+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263370880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.097 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.089+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.097 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.089+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263494a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.104 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.093+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.105 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.093+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622638c6d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.110 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.101+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.110 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.101+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622638c6080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.116 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.105+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.116 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.105+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622638c6400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.122 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.113+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.122 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.113+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622635cb400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.133 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.125+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.133 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.125+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622635cb800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.141 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.129+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.141 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.129+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622635cae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.147 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.137+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.147 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.137+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263577480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.168 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.157+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:00.169 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:13:00.157+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263577300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:13:06.696 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:06.696 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:06.699 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:06.701 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:13:07.677 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/102508650 2023-10-21T04:13:07.678 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:13:07.678 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861587 2023-10-21T04:13:07.679 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:07.679 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/102508650 ]] 2023-10-21T04:13:07.679 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/102508650 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\2\5\0\8\6\5\0 ]] 2023-10-21T04:13:07.679 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:13:17.687 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:17.688 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:17.688 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:17.693 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:13:18.516 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/102508650 2023-10-21T04:13:18.517 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:13:18.518 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861598 2023-10-21T04:13:18.518 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:18.518 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/102508650 ]] 2023-10-21T04:13:18.519 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/102508650 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\2\5\0\8\6\5\0 ]] 2023-10-21T04:13:18.519 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:13:28.520 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:13:28.520 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:28.520 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:28.521 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:29.506 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/102508650 2023-10-21T04:13:29.508 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:13:29.510 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861609 2023-10-21T04:13:29.510 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:29.510 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/102508650 ]] 2023-10-21T04:13:29.510 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/102508650 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\2\5\0\8\6\5\0 ]] 2023-10-21T04:13:29.510 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:13:39.511 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:13:39.512 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:39.521 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:39.527 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:40.397 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/102508650 2023-10-21T04:13:40.398 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:13:40.399 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861620 2023-10-21T04:13:40.399 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:40.400 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/102508650 ]] 2023-10-21T04:13:40.400 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/102508650 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\2\5\0\8\6\5\0 ]] 2023-10-21T04:13:40.400 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:13:50.403 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:13:50.403 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:13:50.405 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:13:50.416 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:13:51.340 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/102508650 2023-10-21T04:13:51.347 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:13:51.350 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861631 2023-10-21T04:13:51.350 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:13:51.350 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/102508650 ]] 2023-10-21T04:13:51.351 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/102508650 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\2\5\0\8\6\5\0 ]] 2023-10-21T04:13:51.351 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:14:00.739 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:14:00.730+0000 7fc58f19b640 -1 reset not still connected to 0x55f03f6768f0 2023-10-21T04:14:00.739 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:14:00.730+0000 7fc58f19b640 -1 reset not still connected to 0x55f044bc9ad0 2023-10-21T04:14:00.739 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:14:00.730+0000 7fc58f19b640 -1 reset not still connected to 0x55f048b84d00 2023-10-21T04:14:00.744 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:14:00.738+0000 7fc58f19b640 -1 reset not still connected to 0x55f05c8cf790 2023-10-21T04:14:01.354 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:01.359 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:14:01.365 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:01.377 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:02.255 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2752721599 2023-10-21T04:14:02.256 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:14:02.257 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861642 2023-10-21T04:14:02.257 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:02.257 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2752721599 ]] 2023-10-21T04:14:02.257 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2752721599 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\2\5\0\8\6\5\0 ]] 2023-10-21T04:14:02.257 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/2752721599 2023-10-21T04:14:03.810 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.802+0000 7fd1f3e25640 -1 librbd::ImageWatcher: 0x562263cca600 image watch failed: 94705704061952, (107) Transport endpoint is not connected 2023-10-21T04:14:03.811 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.802+0000 7fd1f3e25640 -1 librbd::Watcher: 0x562263cca600 handle_error: handle=94705704061952: (107) Transport endpoint is not connected 2023-10-21T04:14:03.811 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.802+0000 7fd1f4626640 -1 librbd::watcher::RewatchRequest: 0x5622631d2640 handle_unwatch client blocklisted 2023-10-21T04:14:03.811 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.802+0000 7fd1f4626640 -1 librbd::Watcher: 0x562263cca600 handle_rewatch: client blocklisted 2023-10-21T04:14:03.832 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.822+0000 7fd1f4626640 -1 librbd::ImageWatcher: 0x562263ccac00 image watch failed: 94705704126464, (107) Transport endpoint is not connected 2023-10-21T04:14:03.833 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.822+0000 7fd1f4626640 -1 librbd::Watcher: 0x562263ccac00 handle_error: handle=94705704126464: (107) Transport endpoint is not connected 2023-10-21T04:14:03.833 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::ImageWatcher: 0x562263ccaf00 image watch failed: 94705701168128, (107) Transport endpoint is not connected 2023-10-21T04:14:03.833 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::Watcher: 0x562263ccaf00 handle_error: handle=94705701168128: (107) Transport endpoint is not connected 2023-10-21T04:14:03.834 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::managed_lock::GetLockerRequest: 0x562262f4c0e0 handle_get_lockers: failed to retrieve lockers: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:03.834 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::managed_lock::AcquireRequest: 0x56226289f950 handle_get_locker: failed to retrieve lockers: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:03.835 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::ManagedLock: 0x5622625ee4f8 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:03.835 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bdba0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:03.835 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::io::AioCompletion: 0x562263db9b80 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:03.836 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::watcher::RewatchRequest: 0x5622635141e0 handle_unwatch client blocklisted 2023-10-21T04:14:03.836 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::Watcher: 0x562263ccac00 handle_rewatch: client blocklisted 2023-10-21T04:14:03.836 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::watcher::RewatchRequest: 0x5622634a2230 handle_unwatch client blocklisted 2023-10-21T04:14:03.836 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::ManagedLock: 0x562262fcf538 send_acquire_lock: watcher not registered - client blocklisted 2023-10-21T04:14:03.836 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bdc70 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:03.837 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:03.826+0000 7fd1f3e25640 -1 librbd::io::AioCompletion: 0x562263da8b00 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:04.704 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/2752721599 until 2023-10-21T05:14:03.862955+0000 (3600 sec) 2023-10-21T04:14:04.711 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:04.702+0000 7fd1f4626640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:04.711 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:04.702+0000 7fd1f4626640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5622632bd930 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:04.711 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:04.702+0000 7fd1f4626640 -1 librbd::io::AioCompletion: 0x562263db8160 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:04.712 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:04.702+0000 7fd1f3e25640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:04.712 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:04.702+0000 7fd1f3e25640 -1 librbd::io::AioCompletion: 0x562263982f20 fail: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:04.712 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:04.702+0000 7fd1f3e25640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x562263affac0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:04.712 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:14:04.702+0000 7fd1f3e25640 -1 librbd::managed_lock::ReleaseRequest: 0x562262a2d320 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:14:04.730 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:14:04.744 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/2752721599 2023-10-21T04:14:05.359 INFO:tasks.workunit.client.0.smithi133.stderr:listed 10 entries 2023-10-21T04:14:05.404 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/2752721599 2023-10-21T04:14:05.404 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:14:15.407 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:15.407 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:14:15.411 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:15.424 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:16.298 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2752721599 2023-10-21T04:14:16.298 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:14:16.299 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861656 2023-10-21T04:14:16.299 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:16.299 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2752721599 ]] 2023-10-21T04:14:16.300 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2752721599 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\7\5\2\7\2\1\5\9\9 ]] 2023-10-21T04:14:16.300 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:14:24.177 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:14:24.158+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:14:24.177 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:14:24.158+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:14:26.301 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:26.301 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:26.302 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:26.324 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:14:27.497 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2752721599 2023-10-21T04:14:27.498 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:14:27.498 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861667 2023-10-21T04:14:27.498 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:27.499 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2752721599 ]] 2023-10-21T04:14:27.499 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2752721599 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\7\5\2\7\2\1\5\9\9 ]] 2023-10-21T04:14:27.499 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:14:37.501 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:37.501 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:14:37.510 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:37.512 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:38.470 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2752721599 2023-10-21T04:14:38.470 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:14:38.471 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861678 2023-10-21T04:14:38.471 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:38.472 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2752721599 ]] 2023-10-21T04:14:38.472 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2752721599 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\7\5\2\7\2\1\5\9\9 ]] 2023-10-21T04:14:38.472 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:14:48.472 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:14:48.472 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:14:48.473 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:48.473 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:49.420 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2752721599 2023-10-21T04:14:49.421 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:14:49.421 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861689 2023-10-21T04:14:49.421 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:14:49.422 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2752721599 ]] 2023-10-21T04:14:49.422 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2752721599 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\7\5\2\7\2\1\5\9\9 ]] 2023-10-21T04:14:49.422 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:14:59.428 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:14:59.429 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:14:59.430 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:14:59.434 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:00.268 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2752721599 2023-10-21T04:15:00.268 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:15:00.269 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861700 2023-10-21T04:15:00.269 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:00.269 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2752721599 ]] 2023-10-21T04:15:00.270 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2752721599 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\7\5\2\7\2\1\5\9\9 ]] 2023-10-21T04:15:00.270 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:15:10.272 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:10.272 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:10.274 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:10.275 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:15:11.253 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2426556961 2023-10-21T04:15:11.253 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:15:11.254 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861711 2023-10-21T04:15:11.254 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:11.255 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2426556961 ]] 2023-10-21T04:15:11.255 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2426556961 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\7\5\2\7\2\1\5\9\9 ]] 2023-10-21T04:15:11.255 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/2426556961 2023-10-21T04:15:13.732 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/2426556961 until 2023-10-21T05:15:12.967952+0000 (3600 sec) 2023-10-21T04:15:13.771 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/2426556961 2023-10-21T04:15:13.778 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:15:14.529 INFO:tasks.workunit.client.0.smithi133.stderr:listed 11 entries 2023-10-21T04:15:14.568 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/2426556961 2023-10-21T04:15:14.569 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:15:24.572 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:24.572 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:24.581 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:24.582 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:15:25.570 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2426556961 2023-10-21T04:15:25.571 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:15:25.571 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861725 2023-10-21T04:15:25.572 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:25.572 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2426556961 ]] 2023-10-21T04:15:25.572 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2426556961 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\4\2\6\5\5\6\9\6\1 ]] 2023-10-21T04:15:25.572 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:15:35.581 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:15:35.581 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:35.583 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:35.586 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:36.448 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2426556961 2023-10-21T04:15:36.449 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:15:36.449 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861736 2023-10-21T04:15:36.449 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:36.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2426556961 ]] 2023-10-21T04:15:36.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2426556961 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\4\2\6\5\5\6\9\6\1 ]] 2023-10-21T04:15:36.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:15:42.705 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:15:42.691+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:15:42.706 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:15:42.699+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:15:42.706 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:15:42.699+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:15:42.707 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:15:42.699+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:15:46.454 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:15:46.454 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:46.455 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:46.477 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:47.362 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2426556961 2023-10-21T04:15:47.362 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:15:47.363 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861747 2023-10-21T04:15:47.363 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:47.363 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2426556961 ]] 2023-10-21T04:15:47.363 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2426556961 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\4\2\6\5\5\6\9\6\1 ]] 2023-10-21T04:15:47.363 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:15:57.379 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:15:57.379 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:15:57.390 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:15:57.395 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:15:58.213 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2426556961 2023-10-21T04:15:58.213 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:15:58.214 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861758 2023-10-21T04:15:58.214 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:15:58.214 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2426556961 ]] 2023-10-21T04:15:58.215 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2426556961 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\4\2\6\5\5\6\9\6\1 ]] 2023-10-21T04:15:58.215 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:16:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.007+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.013 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.007+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.013 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.007+0000 7fd1f4e27640 -1 librbd::ImageState: 0x5622638c6500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.013 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.007+0000 7fd1f4e27640 -1 librbd::ImageState: 0x5622637fe980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.015 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.007+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.015 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.007+0000 7fd1f4e27640 -1 librbd::ImageState: 0x5622637ff080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.026 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.019+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.027 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.019+0000 7fd1f4e27640 -1 librbd::ImageState: 0x5622637ff200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.031 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.023+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.031 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.023+0000 7fd1f4e27640 -1 librbd::ImageState: 0x5622637ff400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.045 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.039+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.046 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.039+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263854100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.049 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.043+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.050 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.043+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263854300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.054 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.047+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.054 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.047+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263854500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.063 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.055+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.063 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.055+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263854700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.066 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.059+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.066 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.059+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263854900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.070 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.063+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.070 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.063+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263854b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.083 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.075+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.083 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.075+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263854d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.087 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.079+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.087 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.079+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263854f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.092 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.083+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.092 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.087+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263855100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.099 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.091+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.099 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.091+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263855300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.109 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.103+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.109 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.103+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263855500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.114 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.107+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.115 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.107+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263855700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.126 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.119+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.126 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.119+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263855900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.130 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.123+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.131 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.123+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263855b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.134 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.127+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:00.134 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:16:00.127+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263855d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:16:08.216 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:16:08.216 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:08.216 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:08.218 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:09.272 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2426556961 2023-10-21T04:16:09.273 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:16:09.273 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861769 2023-10-21T04:16:09.273 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:09.274 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2426556961 ]] 2023-10-21T04:16:09.274 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2426556961 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\4\2\6\5\5\6\9\6\1 ]] 2023-10-21T04:16:09.274 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:16:11.771 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.763+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:16:11.772 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.763+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:16:11.772 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.763+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:16:11.773 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.763+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:16:11.773 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.763+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:16:11.773 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.763+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:16:11.775 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.767+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:16:11.775 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.767+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:16:11.775 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.767+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:16:11.775 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.767+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:16:11.775 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.767+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:16:11.776 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.767+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:16:11.776 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:11.767+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:16:19.284 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:16:19.284 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:19.285 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:19.287 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:20.226 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1019917016 2023-10-21T04:16:20.227 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:16:20.227 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861780 2023-10-21T04:16:20.227 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:20.228 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1019917016 ]] 2023-10-21T04:16:20.228 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1019917016 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\4\2\6\5\5\6\9\6\1 ]] 2023-10-21T04:16:20.228 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/1019917016 2023-10-21T04:16:22.682 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/1019917016 until 2023-10-21T05:16:21.899622+0000 (3600 sec) 2023-10-21T04:16:22.711 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/1019917016 2023-10-21T04:16:22.714 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:16:23.521 INFO:tasks.workunit.client.0.smithi133.stderr:listed 12 entries 2023-10-21T04:16:23.584 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/1019917016 2023-10-21T04:16:23.585 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:16:25.009 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:24.999+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:16:25.009 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:24.999+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:16:25.012 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:16:24.999+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:16:33.588 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:16:33.588 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:33.590 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:33.592 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:34.544 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1019917016 2023-10-21T04:16:34.545 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:16:34.545 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861794 2023-10-21T04:16:34.545 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:34.546 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1019917016 ]] 2023-10-21T04:16:34.546 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1019917016 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\1\9\9\1\7\0\1\6 ]] 2023-10-21T04:16:34.546 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:16:44.550 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:44.550 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:44.554 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:44.564 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:16:45.401 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1019917016 2023-10-21T04:16:45.402 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:16:45.402 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861805 2023-10-21T04:16:45.403 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:45.403 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1019917016 ]] 2023-10-21T04:16:45.403 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1019917016 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\1\9\9\1\7\0\1\6 ]] 2023-10-21T04:16:45.403 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:16:55.405 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:16:55.409 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:16:55.415 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:16:55.416 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:16:56.280 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1019917016 2023-10-21T04:16:56.281 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:16:56.281 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861816 2023-10-21T04:16:56.281 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:16:56.282 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1019917016 ]] 2023-10-21T04:16:56.282 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1019917016 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\1\9\9\1\7\0\1\6 ]] 2023-10-21T04:16:56.282 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:17:00.010 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.003+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.010 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.003+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9c100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.007+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.007+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9c300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.015 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.011+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.016 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.011+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9c500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.019 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.011+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.019 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.011+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9c700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.022 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.015+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.023 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.015+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9c900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.027 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.019+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.027 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.019+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.030 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.023+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.030 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.023+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.036 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.027+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.036 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.027+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9cf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.038 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.031+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.039 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.031+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9d680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.043 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.035+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.043 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.035+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9d500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.045 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.039+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.046 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.039+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9d300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.049 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.043+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.049 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.043+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9cd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.052 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.047+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.052 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.047+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9d900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.054 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.047+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.055 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.047+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9db00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.058 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.051+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.058 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.051+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.062 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.055+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.062 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.055+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d9df00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.064 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.059+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.065 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.059+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad6100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.068 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.063+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.068 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.063+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad6300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.071 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.063+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.071 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.067+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad6500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.075 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.067+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:00.076 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:17:00.067+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad6700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:17:06.288 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:17:06.292 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:06.292 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:06.292 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:07.224 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1019917016 2023-10-21T04:17:07.224 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:17:07.225 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861827 2023-10-21T04:17:07.225 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:07.225 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1019917016 ]] 2023-10-21T04:17:07.226 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1019917016 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\1\9\9\1\7\0\1\6 ]] 2023-10-21T04:17:07.226 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:17:17.227 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:17.228 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:17:17.228 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:17.231 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:18.093 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/1019917016 2023-10-21T04:17:18.093 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:17:18.095 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861838 2023-10-21T04:17:18.095 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:18.095 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/1019917016 ]] 2023-10-21T04:17:18.095 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/1019917016 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\1\9\9\1\7\0\1\6 ]] 2023-10-21T04:17:18.095 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:17:19.960 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:17:19.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:17:19.960 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:17:19.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:17:28.104 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:17:28.107 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:28.116 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:28.123 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:29.023 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3237579704 2023-10-21T04:17:29.023 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:17:29.024 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861849 2023-10-21T04:17:29.024 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:29.024 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3237579704 ]] 2023-10-21T04:17:29.024 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3237579704 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\1\0\1\9\9\1\7\0\1\6 ]] 2023-10-21T04:17:29.024 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/3237579704 2023-10-21T04:17:30.693 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/3237579704 until 2023-10-21T05:17:29.946358+0000 (3600 sec) 2023-10-21T04:17:30.743 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/3237579704 2023-10-21T04:17:30.743 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:17:31.437 INFO:tasks.workunit.client.0.smithi133.stderr:listed 13 entries 2023-10-21T04:17:31.464 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/3237579704 2023-10-21T04:17:31.464 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:17:41.470 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:41.470 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:17:41.471 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:41.475 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:42.442 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3237579704 2023-10-21T04:17:42.443 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:17:42.443 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861862 2023-10-21T04:17:42.444 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:42.444 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3237579704 ]] 2023-10-21T04:17:42.444 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3237579704 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\2\3\7\5\7\9\7\0\4 ]] 2023-10-21T04:17:42.444 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:17:52.445 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:17:52.451 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:17:52.455 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:17:52.455 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:17:53.388 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3237579704 2023-10-21T04:17:53.389 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:17:53.390 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861873 2023-10-21T04:17:53.390 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:17:53.390 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3237579704 ]] 2023-10-21T04:17:53.390 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3237579704 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\2\3\7\5\7\9\7\0\4 ]] 2023-10-21T04:17:53.390 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:18:00.007 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.003+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.008 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.003+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263896e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.011 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.003+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.011 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.003+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263371600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.007+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.007+0000 7fd1f4e27640 -1 librbd::ImageState: 0x5622638c7600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.016 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.011+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.017 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.011+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263675580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.019 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.015+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.020 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.015+0000 7fd1f4e27640 -1 librbd::ImageState: 0x5622638c6a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.022 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.015+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.023 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.015+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad6900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.025 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.019+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.025 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.019+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad6d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.028 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.023+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.028 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.023+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad6c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.031 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.023+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.031 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.023+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263ad7300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.034 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.027+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.035 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.027+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad7180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.037 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.031+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.037 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.031+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad6f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.040 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.035+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.040 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.035+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263577a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.042 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.035+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.042 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.035+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad7580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.044 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.039+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.044 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.039+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263ad7780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.047 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.039+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.047 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.043+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263ad7980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.050 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.043+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.050 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.043+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263ad7b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.052 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.047+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.053 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.047+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad7d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.056 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.051+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.056 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.051+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad7f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.058 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.051+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.059 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.051+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263c7e180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.062 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.055+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:00.062 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:18:00.055+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263c7e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:18:03.392 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:18:03.392 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:03.393 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:03.394 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:04.357 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3237579704 2023-10-21T04:18:04.357 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:18:04.358 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861884 2023-10-21T04:18:04.358 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:04.358 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3237579704 ]] 2023-10-21T04:18:04.358 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3237579704 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\2\3\7\5\7\9\7\0\4 ]] 2023-10-21T04:18:04.359 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:18:14.360 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:14.361 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:14.367 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:14.371 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:18:15.253 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3237579704 2023-10-21T04:18:15.253 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:18:15.254 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861895 2023-10-21T04:18:15.254 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:15.255 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3237579704 ]] 2023-10-21T04:18:15.255 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3237579704 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\2\3\7\5\7\9\7\0\4 ]] 2023-10-21T04:18:15.255 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:18:25.260 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:25.264 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:18:25.265 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:25.271 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:26.419 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3237579704 2023-10-21T04:18:26.420 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:18:26.421 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861906 2023-10-21T04:18:26.421 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:26.422 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3237579704 ]] 2023-10-21T04:18:26.422 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3237579704 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\2\3\7\5\7\9\7\0\4 ]] 2023-10-21T04:18:26.422 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:18:34.256 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:18:34.256 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:18:34.256 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:18:34.256 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:18:34.257 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:18:34.257 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:18:34.257 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:18:34.257 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:18:34.257 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:18:34.258 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:18:34.258 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:18:34.258 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:18:34.258 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:18:34.258 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:34.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:18:36.423 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:36.423 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:36.424 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:36.427 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:18:37.402 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3729849345 2023-10-21T04:18:37.405 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:18:37.406 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861917 2023-10-21T04:18:37.406 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:37.407 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3729849345 ]] 2023-10-21T04:18:37.407 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3729849345 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\2\3\7\5\7\9\7\0\4 ]] 2023-10-21T04:18:37.407 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/3729849345 2023-10-21T04:18:38.807 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:38.799+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:18:38.809 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:18:38.803+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:18:39.779 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/3729849345 until 2023-10-21T05:18:39.040269+0000 (3600 sec) 2023-10-21T04:18:39.807 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:18:39.809 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/3729849345 2023-10-21T04:18:40.606 INFO:tasks.workunit.client.0.smithi133.stderr:listed 14 entries 2023-10-21T04:18:40.635 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/3729849345 2023-10-21T04:18:40.635 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:18:50.653 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:18:50.653 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:18:50.655 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:18:50.665 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:18:51.694 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3729849345 2023-10-21T04:18:51.708 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:18:51.710 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861931 2023-10-21T04:18:51.710 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:18:51.711 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3729849345 ]] 2023-10-21T04:18:51.711 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3729849345 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\7\2\9\8\4\9\3\4\5 ]] 2023-10-21T04:18:51.711 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:19:00.011 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.003+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.003+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263929c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.007+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.007+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622637feb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.018 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.011+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.018 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.011+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263854e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.021 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.015+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.021 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.015+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562262f5dc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.025 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.019+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.025 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.019+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263855f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.030 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.023+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.030 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.023+0000 7fd1f4626640 -1 librbd::ImageState: 0x56226346cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.048 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.043+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.048 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.043+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263495980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.052 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.047+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.053 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.047+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263675580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.055 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.047+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.055 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.047+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263ad6680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.059 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.051+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.059 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.051+0000 7fd1f4626640 -1 librbd::ImageState: 0x56226346cd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.062 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.055+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.062 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.055+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263577280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.065 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.059+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.065 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.059+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263d9d000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.068 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.063+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.068 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.063+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263c7e800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.071 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.063+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.071 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.063+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263ad6400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.074 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.067+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.074 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.067+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263ad6300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.079 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.071+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.080 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.071+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263c7e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.081 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.075+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.081 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.075+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263c7eb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.085 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.079+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.086 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.079+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263c7ed80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.091 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.083+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.091 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.083+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263c7f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.096 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.091+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:00.097 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:19:00.091+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263c7f600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:19:01.744 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:01.757 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:19:01.757 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:01.758 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:02.883 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3729849345 2023-10-21T04:19:02.883 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:19:02.884 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861942 2023-10-21T04:19:02.884 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:02.885 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3729849345 ]] 2023-10-21T04:19:02.885 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3729849345 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\7\2\9\8\4\9\3\4\5 ]] 2023-10-21T04:19:02.885 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:19:12.891 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:12.892 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:19:12.894 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:12.903 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:13.872 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3729849345 2023-10-21T04:19:13.881 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:19:13.882 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861953 2023-10-21T04:19:13.883 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:13.890 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3729849345 ]] 2023-10-21T04:19:13.890 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3729849345 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\7\2\9\8\4\9\3\4\5 ]] 2023-10-21T04:19:13.890 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:19:23.898 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:23.899 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:19:23.907 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:23.910 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:24.822 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3729849345 2023-10-21T04:19:24.823 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:19:24.823 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861964 2023-10-21T04:19:24.823 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:24.824 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3729849345 ]] 2023-10-21T04:19:24.824 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3729849345 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\7\2\9\8\4\9\3\4\5 ]] 2023-10-21T04:19:24.824 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:19:34.825 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:34.826 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:34.827 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:19:34.839 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:35.771 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3729849345 2023-10-21T04:19:35.772 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:19:35.773 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861975 2023-10-21T04:19:35.773 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:35.774 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3729849345 ]] 2023-10-21T04:19:35.774 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3729849345 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\7\2\9\8\4\9\3\4\5 ]] 2023-10-21T04:19:35.774 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:19:45.776 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:19:45.776 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:45.780 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:19:45.786 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:46.715 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2789602076 2023-10-21T04:19:46.716 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:19:46.716 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697861986 2023-10-21T04:19:46.716 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:19:46.717 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2789602076 ]] 2023-10-21T04:19:46.717 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2789602076 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\7\2\9\8\4\9\3\4\5 ]] 2023-10-21T04:19:46.717 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/2789602076 2023-10-21T04:19:48.785 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/2789602076 until 2023-10-21T05:19:47.947961+0000 (3600 sec) 2023-10-21T04:19:48.820 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:19:48.820 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/2789602076 2023-10-21T04:19:49.441 INFO:tasks.workunit.client.0.smithi133.stderr:listed 15 entries 2023-10-21T04:19:49.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/2789602076 2023-10-21T04:19:49.480 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:19:54.655 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:19:54.655 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:19:54.655 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:19:54.656 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:19:54.656 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:19:54.656 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:19:54.656 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:19:54.656 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:19:54.657 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:19:54.657 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:19:54.657 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:19:54.657 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:19:54.657 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:19:54.658 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:19:54.658 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:19:54.658 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:19:54.658 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:19:54.659 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:19:54.659 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:19:54.642+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:19:59.496 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:19:59.498 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:19:59.499 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:19:59.502 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.002+0000 7fd1f4e27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.002+0000 7fd1f4e27640 -1 librbd::ImageState: 0x562263c7ee00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.006+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.013 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.006+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263c7f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.015 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.006+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.015 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.006+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad7a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.021 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.014+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.021 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.014+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263205700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.028 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.022+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.029 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.022+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263ad7880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.033 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.026+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.034 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.026+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263674000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.045 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.038+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.046 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.038+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2e100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.050 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.042+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.050 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.042+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.055 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.046+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.055 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.046+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.060 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.054+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.060 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.054+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.065 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.058+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.065 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.058+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.071 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.062+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.071 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.062+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2eb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.075 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.066+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.075 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.066+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2ed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.079 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.070+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.079 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.070+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2ef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.083 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.074+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.083 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.074+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.091 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.082+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.091 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.082+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.092 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.086+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.092 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.086+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2f500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.095 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.086+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.095 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.086+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2f700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.099 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.090+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.099 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.090+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2f900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.106 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.098+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.107 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:20:00.098+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263d2fb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:20:00.910 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2789602076 2023-10-21T04:20:00.912 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:20:00.913 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862000 2023-10-21T04:20:00.913 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:00.913 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2789602076 ]] 2023-10-21T04:20:00.913 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2789602076 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\7\8\9\6\0\2\0\7\6 ]] 2023-10-21T04:20:00.914 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:20:10.915 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:20:10.915 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:10.922 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:10.925 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:11.834 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2789602076 2023-10-21T04:20:11.835 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:20:11.835 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862011 2023-10-21T04:20:11.835 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:11.836 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2789602076 ]] 2023-10-21T04:20:11.836 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2789602076 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\7\8\9\6\0\2\0\7\6 ]] 2023-10-21T04:20:11.836 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:20:21.840 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:21.843 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:21.843 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:21.849 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:20:22.761 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2789602076 2023-10-21T04:20:22.761 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:20:22.762 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862022 2023-10-21T04:20:22.762 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:22.762 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2789602076 ]] 2023-10-21T04:20:22.762 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2789602076 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\7\8\9\6\0\2\0\7\6 ]] 2023-10-21T04:20:22.763 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:20:24.787 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:20:24.788 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:20:24.788 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:20:24.788 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:20:24.788 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:20:24.789 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:20:24.789 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:20:24.789 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:20:24.789 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:20:24.789 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:20:24.790 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:20:24.790 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:20:24.790 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:20:24.790 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:24.778+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:20:26.673 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:26.666+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:20:26.673 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:26.666+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:20:32.767 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:20:32.768 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:32.775 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:32.780 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:33.658 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2789602076 2023-10-21T04:20:33.658 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:20:33.659 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862033 2023-10-21T04:20:33.659 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:33.659 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2789602076 ]] 2023-10-21T04:20:33.660 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2789602076 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\7\8\9\6\0\2\0\7\6 ]] 2023-10-21T04:20:33.660 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:20:43.666 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:20:43.672 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:43.679 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:43.680 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:44.332 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:44.326+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:20:44.333 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:44.326+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:20:44.338 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:44.330+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:20:44.338 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:44.330+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:20:44.775 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/2789602076 2023-10-21T04:20:44.776 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:20:44.776 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862044 2023-10-21T04:20:44.777 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:44.777 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/2789602076 ]] 2023-10-21T04:20:44.777 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/2789602076 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\7\8\9\6\0\2\0\7\6 ]] 2023-10-21T04:20:44.777 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:20:50.550 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:50.542+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:20:50.553 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:50.546+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:20:50.553 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:20:50.546+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:20:54.786 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:20:54.793 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:20:54.797 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:20:54.804 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:20:55.840 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:20:55.848 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:20:55.849 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862055 2023-10-21T04:20:55.849 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:20:55.849 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:20:55.849 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\2\7\8\9\6\0\2\0\7\6 ]] 2023-10-21T04:20:55.850 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist add 172.21.15.133:0/3061939021 2023-10-21T04:20:57.810 INFO:tasks.workunit.client.0.smithi133.stderr:blocklisting 172.21.15.133:0/3061939021 until 2023-10-21T05:20:57.015508+0000 (3600 sec) 2023-10-21T04:20:57.856 INFO:tasks.workunit.client.0.smithi133.stderr:+ ceph osd blocklist ls 2023-10-21T04:20:57.857 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep -q 172.21.15.133:0/3061939021 2023-10-21T04:20:58.573 INFO:tasks.workunit.client.0.smithi133.stderr:listed 16 entries 2023-10-21T04:20:58.602 INFO:tasks.workunit.client.0.smithi133.stderr:+ PREV_CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:20:58.603 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:21:00.012 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.002+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.013 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.002+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.013 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.002+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622637ffd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.002+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622635cb180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.006+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.014 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.006+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263446280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.018 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.010+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.018 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.010+0000 7fd1f4626640 -1 librbd::ImageState: 0x5622638c7280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.023 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.014+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.023 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.014+0000 7fd1f4626640 -1 librbd::ImageState: 0x562263896b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.028 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.022+0000 7fd1f4626640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.028 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.022+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263ad6500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.032 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.026+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.033 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.026+0000 7fd1f3e25640 -1 librbd::ImageState: 0x5622631cb780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.046 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.038+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.047 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.038+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263928080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.047 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.038+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.047 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.038+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263c7e400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.052 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.046+0000 7fd1f3e25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:00.053 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T04:21:00.046+0000 7fd1f3e25640 -1 librbd::ImageState: 0x562263c7f000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-21T04:21:08.356 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:21:08.350+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:21:08.604 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:21:08.607 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:08.607 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:08.617 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:09.528 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:21:09.529 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:21:09.529 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862069 2023-10-21T04:21:09.529 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:09.530 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:21:09.530 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:21:09.530 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:21:19.545 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:21:19.546 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:19.555 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:19.556 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:20.576 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:21:20.576 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:21:20.583 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862080 2023-10-21T04:21:20.583 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:20.584 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:21:20.584 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:21:20.584 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:21:30.589 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:30.590 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:30.590 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:21:30.594 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:31.730 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:21:31.730 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:21:31.731 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862091 2023-10-21T04:21:31.731 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:31.731 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:21:31.731 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:21:31.732 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:21:41.733 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:21:41.733 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:41.733 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:41.740 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:42.989 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:21:42.989 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:21:42.990 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862102 2023-10-21T04:21:42.990 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:42.991 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:21:42.991 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:21:42.991 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:21:52.995 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:21:52.996 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:21:52.996 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:21:52.998 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:21:53.851 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:21:53.851 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:21:53.852 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862113 2023-10-21T04:21:53.852 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:21:53.852 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:21:53.853 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:21:53.853 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:22:03.861 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:03.867 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:03.871 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:22:03.886 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:04.816 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:22:04.817 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:22:04.818 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862124 2023-10-21T04:22:04.818 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:04.819 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:22:04.819 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:22:04.819 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:22:14.820 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:22:14.821 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:14.824 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:14.825 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:15.257 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:22:15.250+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:22:15.257 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:22:15.250+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:22:15.260 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:22:15.254+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:22:15.260 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:22:15.254+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:22:15.638 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:22:15.639 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:22:15.639 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862135 2023-10-21T04:22:15.640 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:15.640 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:22:15.640 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:22:15.640 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:22:25.644 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:25.647 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:25.649 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:22:25.651 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:26.410 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:22:26.411 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:22:26.412 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862146 2023-10-21T04:22:26.412 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:26.412 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:22:26.413 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:22:26.413 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:22:36.415 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:22:36.415 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:36.415 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:36.428 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:37.327 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:22:37.327 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:22:37.328 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862157 2023-10-21T04:22:37.328 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:37.329 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:22:37.329 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:22:37.329 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:22:47.338 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:47.340 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:47.341 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:47.342 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:22:48.251 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:22:48.251 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:22:48.252 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862168 2023-10-21T04:22:48.252 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:48.252 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:22:48.253 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:22:48.253 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:22:58.262 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:22:58.262 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:22:58.263 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:22:58.273 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:22:59.311 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:22:59.312 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:22:59.313 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862179 2023-10-21T04:22:59.313 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:22:59.314 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:22:59.314 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:22:59.314 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:23:09.315 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:09.316 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:09.316 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:23:09.319 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:10.268 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:23:10.269 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:23:10.269 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862190 2023-10-21T04:23:10.269 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:10.269 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:23:10.270 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:23:10.270 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:23:20.273 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:23:20.274 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:20.275 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:20.281 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:21.276 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:23:21.276 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:23:21.277 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862201 2023-10-21T04:23:21.277 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:21.278 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:23:21.278 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:23:21.278 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:23:31.279 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:23:31.279 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:31.290 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:31.291 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:32.351 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:23:32.352 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:23:32.352 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862212 2023-10-21T04:23:32.353 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:32.353 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:23:32.353 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:23:32.353 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:23:34.146 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:23:34.130+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:23:42.356 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:23:42.359 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:42.376 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:42.377 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:43.342 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:23:43.347 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:23:43.348 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862223 2023-10-21T04:23:43.348 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:43.348 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:23:43.348 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:23:43.349 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:23:49.060 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:23:49.047+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:23:49.060 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:23:49.047+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:23:49.060 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:23:49.047+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:23:49.060 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:23:49.047+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:23:53.355 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:23:53.356 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:23:53.356 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:23:53.358 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:23:54.342 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:23:54.342 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:23:54.342 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862234 2023-10-21T04:23:54.343 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:23:54.343 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:23:54.343 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:23:54.343 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:24:04.345 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:24:04.346 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:04.349 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:04.362 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:05.261 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:24:05.262 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:24:05.262 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862245 2023-10-21T04:24:05.263 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:05.263 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:24:05.263 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:24:05.263 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:24:15.268 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:24:15.271 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:15.278 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:15.280 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:16.156 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:24:16.157 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:24:16.157 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862256 2023-10-21T04:24:16.157 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:16.158 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:24:16.158 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:24:16.158 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:24:22.177 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:24:22.171+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:24:22.181 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:24:22.175+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:24:26.160 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:26.160 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:26.162 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:24:26.166 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:27.132 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:24:27.133 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:24:27.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862267 2023-10-21T04:24:27.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:27.134 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:24:27.135 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:24:27.135 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:24:37.136 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:24:37.144 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:37.145 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:37.151 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:38.252 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:24:38.252 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:24:38.253 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862278 2023-10-21T04:24:38.253 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:38.253 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:24:38.254 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:24:38.254 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:24:48.255 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:24:48.255 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:48.255 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:48.257 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:24:49.223 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:24:49.224 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:24:49.224 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862289 2023-10-21T04:24:49.224 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:24:49.225 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:24:49.225 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:24:49.225 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:24:59.225 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:24:59.226 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:24:59.228 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:24:59.230 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:00.304 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:25:00.305 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:25:00.305 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862300 2023-10-21T04:25:00.305 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:00.306 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:25:00.306 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:25:00.306 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:25:10.310 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:10.311 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:10.314 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:10.331 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:25:11.488 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:25:11.488 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:25:11.489 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862311 2023-10-21T04:25:11.489 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:11.489 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:25:11.489 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:25:11.489 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:25:21.490 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:25:21.491 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:21.491 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:21.501 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:22.535 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:25:22.536 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:25:22.536 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862322 2023-10-21T04:25:22.537 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:22.537 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:25:22.537 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:25:22.537 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:25:32.539 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:32.539 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:32.557 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:32.559 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:25:33.449 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:25:33.449 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:25:33.449 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862333 2023-10-21T04:25:33.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:33.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:25:33.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:25:33.450 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:25:43.451 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:25:43.452 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:43.455 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:43.461 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:44.330 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:25:44.331 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:25:44.331 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862344 2023-10-21T04:25:44.332 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:44.332 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:25:44.332 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:25:44.332 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:25:54.340 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:25:54.340 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:25:54.340 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:25:54.347 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:25:55.214 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:25:55.214 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:25:55.215 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862355 2023-10-21T04:25:55.215 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:25:55.215 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:25:55.215 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:25:55.216 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:25:58.835 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:25:58.807+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:25:58.836 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:25:58.815+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:25:58.853 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:25:58.839+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:25:58.853 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:25:58.839+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:25:58.854 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:25:58.839+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:26:05.217 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:26:05.222 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:05.226 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:05.227 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:06.218 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:26:06.219 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:26:06.219 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862366 2023-10-21T04:26:06.219 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:06.220 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:26:06.220 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:26:06.220 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:26:16.221 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:26:16.221 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:16.222 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:16.227 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:17.159 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:26:17.159 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:26:17.160 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862377 2023-10-21T04:26:17.160 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:17.160 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:26:17.160 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:26:17.160 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:26:27.162 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:27.164 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:27.169 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:26:27.170 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:28.193 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:26:28.193 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:26:28.194 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862388 2023-10-21T04:26:28.194 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:28.194 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:26:28.194 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:26:28.195 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:26:30.608 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:26:30.599+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:26:30.608 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:26:30.599+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:26:36.546 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:26:36.531+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:26:36.580 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:26:36.547+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:26:38.195 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:26:38.197 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:38.198 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:38.222 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:39.068 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:26:39.069 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:26:39.071 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862399 2023-10-21T04:26:39.071 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:39.071 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:26:39.071 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:26:39.071 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:26:49.076 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:26:49.077 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:26:49.083 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:26:49.104 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:26:50.142 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:26:50.142 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:26:50.143 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862410 2023-10-21T04:26:50.143 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:26:50.144 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:26:50.144 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:26:50.144 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:26:55.980 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:26:55.971+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:26:55.980 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:26:55.971+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:26:55.992 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:26:55.979+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:27:00.143 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:27:00.146 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:00.154 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:00.158 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:01.086 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:27:01.087 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:27:01.088 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862421 2023-10-21T04:27:01.088 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:01.088 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:27:01.088 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:27:01.089 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:27:11.090 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:11.092 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:27:11.093 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:11.094 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:12.047 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:27:12.047 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:27:12.048 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862432 2023-10-21T04:27:12.048 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:12.048 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:27:12.048 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:27:12.048 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:27:22.051 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:22.057 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:22.058 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:27:22.082 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:22.929 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:27:22.929 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:27:22.930 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862442 2023-10-21T04:27:22.930 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:22.930 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:27:22.931 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:27:22.931 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:27:32.941 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:32.942 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:32.944 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:32.954 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:27:33.856 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:27:33.858 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:27:33.859 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862453 2023-10-21T04:27:33.859 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:33.859 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:27:33.860 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:27:33.860 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:27:43.865 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:43.868 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:43.868 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:43.871 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:27:44.744 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:27:44.744 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:27:44.745 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862464 2023-10-21T04:27:44.745 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:44.745 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:27:44.746 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:27:44.746 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:27:54.757 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:27:54.758 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:27:54.760 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:27:54.762 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:27:55.761 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:27:55.768 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:27:55.770 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862475 2023-10-21T04:27:55.770 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:27:55.770 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:27:55.771 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:27:55.771 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:28:05.786 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:05.787 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:05.798 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:05.799 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:28:06.760 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:28:06.760 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:28:06.761 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862486 2023-10-21T04:28:06.761 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:06.761 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:28:06.761 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:28:06.761 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:28:16.773 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:28:16.774 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:16.774 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:16.774 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:17.644 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:28:17.645 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:28:17.645 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862497 2023-10-21T04:28:17.646 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:17.646 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:28:17.646 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:28:17.646 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:28:27.654 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:28:27.663 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:27.665 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:27.673 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:28.643 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:28:28.648 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:28:28.649 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862508 2023-10-21T04:28:28.650 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:28.650 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:28:28.650 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:28:28.650 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:28:38.659 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:28:38.659 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:38.660 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:38.700 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:39.584 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:28:39.585 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:28:39.586 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862519 2023-10-21T04:28:39.586 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:39.587 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:28:39.587 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:28:39.587 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:28:49.588 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:28:49.588 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:28:49.588 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:28:49.589 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:28:50.633 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:28:50.634 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:28:50.634 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862530 2023-10-21T04:28:50.635 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:28:50.635 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:28:50.635 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:28:50.635 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:29:00.651 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:00.652 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:00.668 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:00.685 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:29:01.525 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:29:01.525 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:29:01.526 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862541 2023-10-21T04:29:01.526 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:01.526 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:29:01.527 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:29:01.527 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:29:06.888 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:06.876+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:29:11.528 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:29:11.533 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:11.534 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:11.542 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:12.454 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:29:12.455 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:29:12.455 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862552 2023-10-21T04:29:12.455 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:12.456 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:29:12.456 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:29:12.456 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:29:22.469 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:29:22.469 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:22.470 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:22.483 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:23.422 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:29:23.423 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:29:23.423 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862563 2023-10-21T04:29:23.423 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:23.424 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:29:23.424 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:29:23.424 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:29:27.396 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:27.388+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:29:27.396 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:27.388+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:29:27.411 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:27.404+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:29:27.411 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:27.404+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:29:30.443 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:30.420+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:29:30.443 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:30.420+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:29:30.443 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:30.420+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:29:30.444 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:30.428+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:29:30.444 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:30.432+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:29:30.444 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:30.432+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:29:33.431 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:29:33.434 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:33.435 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:33.435 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:34.366 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:29:34.366 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:29:34.370 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862574 2023-10-21T04:29:34.370 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:34.371 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:29:34.371 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:29:34.371 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:29:34.758 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:34.752+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:29:34.759 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:34.752+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:29:34.759 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:34.752+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:29:34.766 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:34.756+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:29:34.767 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:34.756+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:29:34.772 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:34.760+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:29:34.772 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:34.760+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:29:34.773 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:34.760+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:29:34.773 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:34.760+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:29:44.384 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:29:44.391 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:44.392 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:44.401 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:45.410 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:29:45.411 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:29:45.418 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862585 2023-10-21T04:29:45.418 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:45.418 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:29:45.418 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:29:45.419 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:29:48.727 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:48.708+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:29:48.733 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:48.724+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:29:48.733 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:48.724+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:29:48.733 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:48.724+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:29:48.733 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:48.724+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:29:55.424 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:29:55.434 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:29:55.442 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:29:55.447 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:29:56.270 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:29:56.270 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:29:56.271 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862596 2023-10-21T04:29:56.271 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:29:56.271 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:29:56.271 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:29:56.271 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:29:58.481 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:58.472+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:29:58.486 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:58.476+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:29:58.486 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:58.476+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:29:58.487 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:29:58.476+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:30:06.272 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:30:06.273 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:06.273 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:06.286 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:07.247 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:30:07.248 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:30:07.249 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862607 2023-10-21T04:30:07.249 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:07.250 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:30:07.250 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:30:07.250 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:30:17.250 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:30:17.251 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:17.251 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:17.256 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:18.121 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:30:18.121 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:30:18.122 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862618 2023-10-21T04:30:18.122 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:18.122 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:30:18.122 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:30:18.123 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:30:28.124 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:28.131 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:30:28.134 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:28.135 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:29.009 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:30:29.009 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:30:29.010 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862629 2023-10-21T04:30:29.010 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:29.010 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:30:29.011 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:30:29.011 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:30:39.012 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:39.013 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:39.014 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:39.016 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:30:39.924 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:30:39.924 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:30:39.925 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862639 2023-10-21T04:30:39.925 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:39.925 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:30:39.925 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:30:39.926 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:30:40.793 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:30:40.784+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:30:40.794 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:30:40.784+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:30:40.794 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:30:40.784+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:30:40.794 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:30:40.784+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:30:40.794 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:30:40.784+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:30:49.935 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:30:49.935 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:30:49.936 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:30:49.948 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:30:50.883 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:30:50.883 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:30:50.884 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862650 2023-10-21T04:30:50.884 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:30:50.884 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:30:50.885 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:30:50.885 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:31:00.886 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:00.892 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:31:00.898 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:00.913 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:01.688 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:31:01.688 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:31:01.689 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862661 2023-10-21T04:31:01.689 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:01.689 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:31:01.689 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:31:01.690 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:31:11.691 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:11.691 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:11.693 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:11.707 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:31:12.648 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:31:12.648 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:31:12.655 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862672 2023-10-21T04:31:12.655 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:12.655 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:31:12.655 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:31:12.656 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:31:22.657 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:22.657 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:22.657 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:22.661 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:31:23.613 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:31:23.614 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:31:23.619 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862683 2023-10-21T04:31:23.620 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:23.620 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:31:23.620 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:31:23.620 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:31:25.080 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:31:25.060+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:31:25.081 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:31:25.064+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:31:25.081 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:31:25.064+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:31:25.081 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:31:25.068+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:31:33.622 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:33.622 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:31:33.622 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:33.638 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:34.581 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:31:34.581 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:31:34.582 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862694 2023-10-21T04:31:34.582 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:34.582 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:31:34.582 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:31:34.583 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:31:44.583 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:44.585 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:31:44.586 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:44.589 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:45.560 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:31:45.561 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:31:45.561 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862705 2023-10-21T04:31:45.561 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:45.562 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:31:45.562 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:31:45.562 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:31:46.382 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:31:46.372+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:31:46.382 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:31:46.372+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:31:46.382 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:31:46.372+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:31:46.389 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:31:46.380+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:31:55.563 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:31:55.563 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:31:55.564 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:31:55.587 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:31:56.480 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:31:56.481 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:31:56.482 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862716 2023-10-21T04:31:56.482 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:31:56.482 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:31:56.483 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:31:56.483 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:32:06.484 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:32:06.484 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:06.484 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:06.485 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:07.328 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:32:07.329 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:32:07.329 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862727 2023-10-21T04:32:07.329 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:07.329 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:32:07.330 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:32:07.330 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:32:12.059 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:32:12.052+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:32:12.059 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:32:12.052+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:32:12.063 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:32:12.056+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:32:17.335 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:17.336 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:17.336 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:32:17.342 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:18.372 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:32:18.372 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:32:18.373 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862738 2023-10-21T04:32:18.373 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:18.373 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:32:18.373 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:32:18.374 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:32:28.375 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:28.375 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:28.376 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:28.378 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:32:29.262 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:32:29.262 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:32:29.263 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862749 2023-10-21T04:32:29.263 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:29.263 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:32:29.264 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:32:29.264 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:32:39.265 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:32:39.265 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:39.270 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:39.271 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:40.147 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:32:40.148 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:32:40.148 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862760 2023-10-21T04:32:40.149 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:40.149 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:32:40.149 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:32:40.149 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:32:50.152 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:32:50.153 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:32:50.153 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:32:50.161 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:32:51.109 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:32:51.109 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:32:51.110 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862771 2023-10-21T04:32:51.110 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:32:51.110 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:32:51.110 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:32:51.111 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:33:01.112 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:01.117 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:01.120 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:33:01.129 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:01.980 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:33:01.981 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:33:01.981 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862781 2023-10-21T04:33:01.982 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:01.982 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:33:01.982 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:33:01.982 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:33:11.987 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:11.993 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:33:11.998 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:12.014 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:13.028 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:33:13.028 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:33:13.028 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862793 2023-10-21T04:33:13.029 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:13.029 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:33:13.029 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:33:13.029 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:33:16.759 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:16.753+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:33:16.767 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:16.761+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:33:23.030 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:33:23.031 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:23.031 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:23.031 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:23.996 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:33:23.997 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:33:23.997 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862803 2023-10-21T04:33:23.997 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:23.997 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:33:23.998 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:33:23.998 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:33:33.997 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:33.998 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:34.003 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:34.012 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:33:34.821 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:33:34.822 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:33:34.822 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862814 2023-10-21T04:33:34.823 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:34.823 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:33:34.823 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:33:34.823 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:33:44.824 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:33:44.825 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:44.827 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:44.828 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:45.675 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:33:45.675 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:33:45.675 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862825 2023-10-21T04:33:45.676 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:45.676 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:33:45.676 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:33:45.676 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:33:49.285 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:33:49.285 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:33:49.285 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:33:49.286 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:33:49.286 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:33:49.286 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:33:49.286 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:33:49.286 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:33:49.287 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:33:49.287 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:33:49.287 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:33:49.287 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:33:49.287 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:33:49.288 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:33:49.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:33:55.683 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:33:55.684 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:33:55.691 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:33:55.699 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:33:56.526 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:33:56.526 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:33:56.527 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862836 2023-10-21T04:33:56.527 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:33:56.527 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:33:56.527 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:33:56.528 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:34:06.529 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:06.531 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:34:06.535 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:06.536 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:07.495 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:34:07.495 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:34:07.496 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862847 2023-10-21T04:34:07.496 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:07.496 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:34:07.496 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:34:07.497 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:34:17.504 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:17.505 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:17.510 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:34:17.523 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:18.434 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:34:18.434 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:34:18.435 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862858 2023-10-21T04:34:18.435 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:18.435 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:34:18.435 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:34:18.436 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:34:28.437 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:28.438 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:28.439 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:34:28.459 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:29.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:34:29.480 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:34:29.481 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862869 2023-10-21T04:34:29.481 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:29.481 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:34:29.481 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:34:29.481 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:34:39.491 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:34:39.492 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:39.497 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:39.515 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:40.436 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:34:40.437 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:34:40.437 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862880 2023-10-21T04:34:40.438 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:40.438 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:34:40.438 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:34:40.438 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:34:50.450 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:34:50.451 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:34:50.451 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:34:50.456 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:34:51.382 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:34:51.382 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:34:51.383 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862891 2023-10-21T04:34:51.383 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:34:51.383 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:34:51.383 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:34:51.384 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:35:01.396 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:35:01.396 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:01.400 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:01.401 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:02.297 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:35:02.297 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:35:02.298 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862902 2023-10-21T04:35:02.298 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:02.298 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:35:02.299 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:35:02.299 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:35:12.176 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:12.169+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:35:12.180 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:12.173+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:35:12.180 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:12.173+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:35:12.181 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:12.173+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:35:12.181 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:12.173+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:35:12.181 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:12.177+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:35:12.199 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:12.193+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:35:12.206 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:12.193+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:35:12.206 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:12.193+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:35:12.207 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:12.193+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:35:12.300 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:35:12.300 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:12.319 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:12.329 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:13.128 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:35:13.128 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:35:13.129 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862913 2023-10-21T04:35:13.129 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:13.129 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:35:13.129 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:35:13.130 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:35:14.756 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:35:14.757 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:35:14.757 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:35:14.757 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:35:14.757 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:35:14.758 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:35:14.758 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:35:14.758 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:35:14.758 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:35:14.758 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:35:14.759 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:35:14.759 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:35:14.760 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:35:14.760 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:14.749+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:35:23.135 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:35:23.145 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:23.146 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:23.156 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:24.084 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:35:24.084 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:35:24.085 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862924 2023-10-21T04:35:24.085 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:24.085 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:35:24.086 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:35:24.086 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:35:34.087 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:34.088 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:34.088 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:34.097 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:35:35.065 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:35:35.065 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:35:35.066 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862935 2023-10-21T04:35:35.066 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:35.067 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:35:35.067 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:35:35.067 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:35:41.191 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:41.181+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:35:41.192 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:41.185+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:35:41.193 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:41.185+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:35:41.193 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:41.185+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:35:45.068 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:35:45.068 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:45.073 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:45.084 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:45.957 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:35:45.958 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:35:45.958 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862945 2023-10-21T04:35:45.958 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:45.959 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:35:45.959 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:35:45.959 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:35:55.968 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:35:55.969 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:35:55.972 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:35:56.002 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:35:56.267 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:56.261+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:35:56.268 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:56.261+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:35:56.269 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:56.261+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:35:56.283 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:56.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:35:56.283 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:35:56.277+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:35:56.940 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:35:56.941 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:35:56.941 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862956 2023-10-21T04:35:56.941 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:35:56.942 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:35:56.942 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:35:56.942 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:36:03.903 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:36:03.897+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:36:06.943 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:36:06.944 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:06.944 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:06.944 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:07.872 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:36:07.873 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:36:07.873 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862967 2023-10-21T04:36:07.874 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:07.874 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:36:07.874 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:36:07.874 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:36:17.876 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:36:17.877 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:17.880 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:17.882 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:18.811 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:36:18.812 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:36:18.812 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862978 2023-10-21T04:36:18.812 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:18.813 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:36:18.813 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:36:18.813 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:36:28.816 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:36:28.817 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:28.817 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:28.821 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:29.710 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:36:29.711 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:36:29.711 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697862989 2023-10-21T04:36:29.712 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:29.712 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:36:29.712 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:36:29.712 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:36:39.715 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:39.715 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:39.718 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:36:39.741 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:40.597 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:36:40.597 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:36:40.598 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863000 2023-10-21T04:36:40.598 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:40.598 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:36:40.598 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:36:40.599 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:36:50.601 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:36:50.602 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:36:50.611 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:36:50.628 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:36:51.572 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:36:51.576 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:36:51.577 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863011 2023-10-21T04:36:51.577 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:36:51.578 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:36:51.578 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:36:51.578 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:37:01.584 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:37:01.586 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:01.597 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:01.613 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:02.459 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:37:02.460 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:37:02.460 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863022 2023-10-21T04:37:02.460 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:02.460 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:37:02.461 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:37:02.461 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:37:10.483 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:37:10.483 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:37:10.483 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:37:10.483 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:37:10.484 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:37:10.484 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:37:10.484 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:37:10.484 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:37:10.485 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:37:10.485 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:37:10.485 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:37:10.485 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:37:10.485 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:10.477+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:37:12.461 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:12.462 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:12.473 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:37:12.481 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:13.517 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:37:13.518 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:37:13.519 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863033 2023-10-21T04:37:13.519 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:13.520 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:37:13.520 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:37:13.520 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:37:18.696 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:18.689+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:37:21.065 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:21.053+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:37:22.411 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:22.405+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:37:22.411 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:22.405+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:37:22.414 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:22.409+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:37:22.421 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:37:22.417+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:37:23.520 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:37:23.520 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:23.521 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:23.526 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:24.380 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:37:24.380 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:37:24.381 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863044 2023-10-21T04:37:24.382 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:24.382 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:37:24.382 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:37:24.382 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:37:34.384 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:34.385 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:34.394 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:34.401 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:37:35.440 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:37:35.441 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:37:35.441 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863055 2023-10-21T04:37:35.441 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:35.442 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:37:35.442 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:37:35.442 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:37:45.444 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:37:45.444 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:45.445 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:45.458 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:46.424 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:37:46.425 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:37:46.425 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863066 2023-10-21T04:37:46.426 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:46.426 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:37:46.426 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:37:46.426 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:37:56.433 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:37:56.434 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:37:56.439 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:37:56.445 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:37:57.445 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:37:57.445 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:37:57.446 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863077 2023-10-21T04:37:57.446 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:37:57.446 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:37:57.446 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:37:57.447 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:38:07.448 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:38:07.448 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:07.449 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:07.456 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:08.364 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:38:08.365 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:38:08.365 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863088 2023-10-21T04:38:08.366 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:08.366 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:38:08.366 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:38:08.366 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:38:18.367 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:38:18.368 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:18.368 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:18.379 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:19.230 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:38:19.230 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:38:19.231 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863099 2023-10-21T04:38:19.231 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:19.231 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:38:19.231 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:38:19.232 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:38:29.233 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:38:29.234 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:29.239 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:29.253 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:30.211 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:38:30.213 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:38:30.213 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863110 2023-10-21T04:38:30.214 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:30.214 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:38:30.214 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:38:30.214 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:38:40.217 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:38:40.218 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:40.218 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:40.225 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:41.072 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:38:41.073 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:38:41.074 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863121 2023-10-21T04:38:41.074 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:41.074 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:38:41.074 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:38:41.074 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:38:47.522 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:47.512+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:38:47.524 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:47.516+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:38:51.076 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:38:51.076 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:38:51.076 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:38:51.098 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:38:51.966 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:38:51.967 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:38:51.967 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863131 2023-10-21T04:38:51.967 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:38:51.968 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:38:51.968 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:38:51.968 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:38:54.653 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:54.644+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:38:54.665 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:54.656+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:38:54.665 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:54.656+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:38:54.666 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:54.656+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:38:54.666 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:54.656+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:38:54.680 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:54.672+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:38:54.680 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:54.672+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:38:54.680 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:54.672+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:38:54.680 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:54.672+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:38:54.681 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:54.672+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:38:54.681 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:54.672+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:38:54.681 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:38:54.672+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:39:00.242 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:39:00.232+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:39:00.254 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:39:00.248+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:39:00.255 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:39:00.248+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:39:01.968 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:39:01.969 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:01.969 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:01.985 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:02.939 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:39:02.939 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:39:02.940 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863142 2023-10-21T04:39:02.940 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:02.940 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:39:02.940 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:39:02.940 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:39:12.945 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:39:12.946 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:12.961 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:12.964 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:13.857 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:39:13.857 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:39:13.858 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863153 2023-10-21T04:39:13.858 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:13.859 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:39:13.859 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:39:13.859 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:39:19.782 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:39:19.772+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:39:19.782 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:39:19.772+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:39:19.791 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:39:19.780+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:39:23.861 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:39:23.865 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:23.867 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:23.871 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:24.777 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:39:24.778 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:39:24.778 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863164 2023-10-21T04:39:24.779 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:24.779 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:39:24.779 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:39:24.779 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:39:34.780 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:39:34.781 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:34.781 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:34.787 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:35.789 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:39:35.790 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:39:35.790 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863175 2023-10-21T04:39:35.790 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:35.791 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:39:35.791 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:39:35.791 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:39:45.794 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:39:45.796 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:45.798 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:45.801 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:46.721 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:39:46.725 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:39:46.727 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863186 2023-10-21T04:39:46.727 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:46.727 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:39:46.727 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:39:46.727 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:39:56.730 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:39:56.731 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:39:56.735 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:39:56.745 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:39:57.670 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:39:57.671 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:39:57.671 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863197 2023-10-21T04:39:57.672 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:39:57.672 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:39:57.672 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:39:57.672 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:40:07.675 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:40:07.676 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:07.678 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:07.684 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:08.766 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:40:08.766 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:40:08.767 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863208 2023-10-21T04:40:08.767 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:08.767 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:40:08.767 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:40:08.767 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:40:18.770 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:40:18.772 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:18.775 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:18.797 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:19.739 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:40:19.739 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:40:19.740 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863219 2023-10-21T04:40:19.740 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:19.740 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:40:19.741 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:40:19.741 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:40:23.545 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:23.536+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:40:23.558 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:23.548+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:40:23.559 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:23.548+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:40:23.569 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:23.560+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:40:23.570 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:23.560+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:40:29.253 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:29.244+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:40:29.267 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:29.260+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:40:29.267 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:29.260+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:40:29.267 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:29.260+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:40:29.741 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:29.742 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:40:29.751 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:29.770 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:30.566 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:40:30.567 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:40:30.568 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863230 2023-10-21T04:40:30.568 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:30.569 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:40:30.569 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:40:30.569 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:40:40.569 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:40.570 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:40.575 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:40.585 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:40:41.373 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.364+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:40:41.377 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.372+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:40:41.377 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.372+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:40:41.378 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.372+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:40:41.378 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.372+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:40:41.378 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.372+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:40:41.381 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.376+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:40:41.381 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.376+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:40:41.382 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.376+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:40:41.384 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.376+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:40:41.386 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.380+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:40:41.386 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.380+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:40:41.395 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:41.388+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:40:41.634 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:40:41.635 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:40:41.635 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863241 2023-10-21T04:40:41.635 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:41.636 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:40:41.636 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:40:41.636 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:40:51.638 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:40:51.638 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:40:51.640 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:40:51.642 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:40:52.600 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:40:52.600 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:40:52.601 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863252 2023-10-21T04:40:52.601 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:40:52.602 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:40:52.602 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:40:52.602 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:40:56.583 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:56.576+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:40:56.597 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:56.592+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:40:56.597 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:56.592+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:40:56.598 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:56.592+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:40:56.598 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:56.592+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:40:56.609 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:56.600+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:40:56.609 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:56.600+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:40:56.609 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:56.604+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:40:56.610 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:56.604+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:40:56.613 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:56.608+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:40:56.613 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:56.608+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:40:56.614 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:40:56.608+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:41:02.607 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:41:02.607 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:02.609 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:02.629 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:03.487 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:41:03.487 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:41:03.491 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863263 2023-10-21T04:41:03.491 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:03.491 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:41:03.491 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:41:03.491 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:41:13.492 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:41:13.493 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:13.493 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:13.499 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:14.364 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:41:14.365 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:41:14.365 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863274 2023-10-21T04:41:14.365 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:14.366 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:41:14.366 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:41:14.366 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:41:24.369 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:24.369 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:24.373 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:41:24.394 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:25.248 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:41:25.254 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:41:25.255 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863285 2023-10-21T04:41:25.256 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:25.256 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:41:25.256 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:41:25.256 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:41:35.263 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:35.263 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:35.264 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:41:35.268 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:36.230 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:41:36.230 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:41:36.231 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863296 2023-10-21T04:41:36.231 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:36.231 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:41:36.231 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:41:36.232 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:41:46.241 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:41:46.241 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:46.246 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:46.266 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:47.215 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:41:47.215 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:41:47.216 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863307 2023-10-21T04:41:47.216 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:47.216 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:41:47.217 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:41:47.217 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:41:57.221 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:41:57.221 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:41:57.226 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:41:57.236 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:41:58.164 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:41:58.164 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:41:58.165 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863318 2023-10-21T04:41:58.165 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:41:58.165 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:41:58.165 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:41:58.166 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:42:08.169 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:08.172 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:08.173 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:42:08.178 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:09.105 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:42:09.105 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:42:09.106 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863329 2023-10-21T04:42:09.106 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:09.106 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:42:09.107 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:42:09.107 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:42:19.108 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:42:19.108 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:19.108 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:19.109 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:19.905 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:42:19.905 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:42:19.906 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863339 2023-10-21T04:42:19.906 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:19.906 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:42:19.906 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:42:19.907 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:42:20.653 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:20.644+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:42:20.669 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:20.664+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:42:20.670 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:20.664+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:42:20.674 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:20.668+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:42:29.907 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:42:29.907 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:29.913 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:29.939 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:30.853 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:42:30.853 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:42:30.853 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863350 2023-10-21T04:42:30.854 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:30.854 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:42:30.854 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:42:30.854 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:42:35.903 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:35.896+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:42:35.903 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:35.896+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:42:35.903 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:35.896+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:42:35.911 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:35.904+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:42:35.919 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:35.912+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:42:35.930 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:35.924+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:42:35.930 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:35.924+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:42:36.453 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:36.444+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:42:36.453 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:36.444+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:42:36.453 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:36.444+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:42:36.460 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:36.452+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:42:36.461 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:36.452+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:42:36.465 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:36.456+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:42:36.478 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:36.468+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:42:36.478 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:36.468+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:42:40.856 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:42:40.857 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:40.858 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:40.868 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:41.775 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:42:41.775 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:42:41.776 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863361 2023-10-21T04:42:41.776 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:41.776 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:42:41.776 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:42:41.777 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:42:51.782 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:42:51.783 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:42:51.783 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:42:51.784 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:42:52.792 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:42:52.792 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:42:52.793 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863372 2023-10-21T04:42:52.793 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:42:52.794 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:42:52.794 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:42:52.794 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:42:56.025 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:56.016+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:42:56.026 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:56.020+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:42:56.027 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:56.020+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:42:56.027 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:42:56.020+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:43:02.796 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:02.808 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:43:02.812 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:02.818 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:03.703 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:43:03.703 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:43:03.704 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863383 2023-10-21T04:43:03.704 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:03.704 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:43:03.705 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:43:03.705 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:43:10.506 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:43:10.496+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:43:10.507 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:43:10.496+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:43:13.706 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:43:13.706 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:13.706 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:13.707 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:14.612 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:43:14.612 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:43:14.613 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863394 2023-10-21T04:43:14.613 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:14.614 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:43:14.614 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:43:14.614 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:43:24.616 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:24.619 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:24.620 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:24.627 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:43:25.459 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:43:25.459 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:43:25.460 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863405 2023-10-21T04:43:25.460 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:25.460 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:43:25.460 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:43:25.461 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:43:35.464 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:35.464 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:35.464 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:35.472 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:43:36.285 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:43:36.291 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:43:36.292 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863416 2023-10-21T04:43:36.293 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:36.293 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:43:36.293 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:43:36.293 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:43:46.310 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:43:46.310 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:46.317 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:46.318 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:47.255 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:43:47.256 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:43:47.256 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863427 2023-10-21T04:43:47.257 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:47.257 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:43:47.257 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:43:47.257 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:43:57.258 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:43:57.258 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:43:57.259 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:43:57.261 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:43:58.232 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:43:58.232 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:43:58.233 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863438 2023-10-21T04:43:58.233 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:43:58.233 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:43:58.233 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:43:58.234 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:44:08.234 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:44:08.235 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:08.245 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:08.252 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:09.029 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:44:09.029 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:44:09.030 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863449 2023-10-21T04:44:09.030 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:09.031 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:44:09.031 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:44:09.031 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:44:10.230 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:10.220+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:44:10.231 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:10.224+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:44:10.232 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:10.224+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:44:10.236 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:10.228+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:44:19.038 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:19.043 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:19.049 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:19.058 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:44:19.957 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:44:19.957 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:44:19.958 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863459 2023-10-21T04:44:19.959 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:19.959 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:44:19.959 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:44:19.959 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:44:29.968 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:29.971 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:44:29.976 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:29.977 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:30.938 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:44:30.941 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:44:30.942 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863470 2023-10-21T04:44:30.942 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:30.942 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:44:30.943 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:44:30.943 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:44:40.944 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:44:40.944 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:40.944 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:40.946 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:41.374 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:41.364+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:44:41.380 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:41.368+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:44:41.504 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:41.496+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:44:41.537 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:41.528+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:44:41.537 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:41.528+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:44:41.911 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:44:41.912 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:44:41.912 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863481 2023-10-21T04:44:41.912 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:41.912 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:44:41.912 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:44:41.913 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:44:51.914 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:44:51.915 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:44:51.915 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:44:51.920 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:44:52.838 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:44:52.838 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:44:52.839 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863492 2023-10-21T04:44:52.839 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:44:52.839 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:44:52.839 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:44:52.840 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:44:58.098 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:44:58.098 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:44:58.098 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:44:58.099 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:44:58.099 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:44:58.099 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:44:58.099 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:44:58.099 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:44:58.100 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:44:58.100 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:44:58.100 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:44:58.100 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:44:58.100 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:44:58.101 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:44:58.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:45:02.841 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:45:02.841 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:02.843 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:02.843 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:03.800 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:45:03.800 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:45:03.800 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863503 2023-10-21T04:45:03.801 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:03.801 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:45:03.801 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:45:03.801 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:45:09.103 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:45:09.096+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:45:09.104 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:45:09.096+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:45:13.803 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:45:13.804 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:13.805 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:13.806 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:14.706 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:45:14.707 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:45:14.708 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863514 2023-10-21T04:45:14.708 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:14.708 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:45:14.709 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:45:14.709 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:45:24.710 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:24.711 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:24.716 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:45:24.717 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:25.773 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:45:25.773 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:45:25.774 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863525 2023-10-21T04:45:25.774 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:25.774 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:45:25.775 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:45:25.775 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:45:35.776 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:35.780 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:35.781 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:35.782 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:45:36.786 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:45:36.787 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:45:36.787 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863536 2023-10-21T04:45:36.787 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:36.787 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:45:36.788 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:45:36.788 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:45:46.792 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:45:46.794 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:46.796 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:46.798 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:47.702 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:45:47.707 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:45:47.708 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863547 2023-10-21T04:45:47.708 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:47.708 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:45:47.709 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:45:47.709 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:45:57.710 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:45:57.711 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:45:57.711 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:45:57.712 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:45:58.565 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:45:58.566 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:45:58.566 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863558 2023-10-21T04:45:58.566 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:45:58.566 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:45:58.567 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:45:58.567 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:46:05.696 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:46:05.688+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:46:05.700 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:46:05.692+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:46:05.724 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:46:05.712+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:46:08.567 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:46:08.567 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:08.567 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:08.570 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:09.468 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:46:09.468 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:46:09.469 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863569 2023-10-21T04:46:09.469 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:09.469 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:46:09.470 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:46:09.470 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:46:19.475 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:46:19.475 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:19.478 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:19.492 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:20.302 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:46:20.303 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:46:20.303 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863580 2023-10-21T04:46:20.304 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:20.304 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:46:20.304 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:46:20.304 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:46:30.306 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:30.306 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:30.313 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:30.329 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:46:31.317 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:46:31.318 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:46:31.318 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863591 2023-10-21T04:46:31.319 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:31.319 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:46:31.319 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:46:31.319 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:46:41.332 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:46:41.332 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:41.344 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:41.346 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:42.277 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:46:42.278 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:46:42.278 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863602 2023-10-21T04:46:42.278 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:42.278 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:46:42.278 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:46:42.279 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:46:52.293 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:46:52.293 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:46:52.293 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:46:52.293 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:46:53.234 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:46:53.234 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:46:53.235 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863613 2023-10-21T04:46:53.235 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:46:53.236 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:46:53.236 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:46:53.236 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:47:03.240 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:03.241 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:47:03.254 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:03.256 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:03.670 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:03.656+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:47:03.671 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:03.656+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:47:03.671 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:03.656+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:47:03.671 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:03.656+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:47:03.676 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:03.668+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:47:03.697 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:03.688+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:47:03.701 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:03.692+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:47:03.701 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:03.692+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:47:04.188 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:47:04.188 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:47:04.189 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863624 2023-10-21T04:47:04.189 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:04.189 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:47:04.190 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:47:04.190 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:47:10.744 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:10.732+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:47:14.193 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:14.193 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:47:14.205 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:14.206 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:15.049 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:47:15.049 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:47:15.050 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863635 2023-10-21T04:47:15.050 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:15.050 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:47:15.050 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:47:15.051 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:47:25.052 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:47:25.052 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:25.052 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:25.053 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:26.000 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:47:26.003 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:47:26.004 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863646 2023-10-21T04:47:26.004 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:26.004 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:47:26.005 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:47:26.005 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:47:36.006 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:47:36.006 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:36.013 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:36.025 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:36.873 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:47:36.873 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:47:36.874 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863656 2023-10-21T04:47:36.874 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:36.874 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:47:36.875 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:47:36.875 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:47:42.258 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:42.251+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:47:42.268 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:42.259+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:47:42.269 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:42.259+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:47:42.281 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:47:42.271+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:47:46.875 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:47:46.876 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:46.898 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:46.909 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:47.776 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:47:47.776 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:47:47.777 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863667 2023-10-21T04:47:47.777 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:47.777 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:47:47.778 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:47:47.778 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:47:57.784 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:47:57.786 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:47:57.792 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:47:57.797 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:47:58.715 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:47:58.715 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:47:58.716 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863678 2023-10-21T04:47:58.717 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:47:58.717 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:47:58.717 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:47:58.717 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:48:08.718 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:48:08.718 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:08.725 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:08.730 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:09.831 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:48:09.835 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:48:09.839 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863689 2023-10-21T04:48:09.839 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:09.840 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:48:09.840 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:48:09.840 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:48:13.286 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:13.275+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:48:13.290 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:13.283+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:48:13.290 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:13.283+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:48:19.841 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:48:19.841 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:19.842 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:19.842 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:20.777 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:48:20.777 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:48:20.778 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863700 2023-10-21T04:48:20.778 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:20.778 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:48:20.778 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:48:20.779 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:48:30.782 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:30.783 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:30.790 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:48:30.791 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:31.692 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:48:31.696 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:48:31.697 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863711 2023-10-21T04:48:31.697 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:31.697 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:48:31.697 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:48:31.698 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:48:38.820 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:38.807+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:48:38.821 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:38.807+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:48:41.358 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.351+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:48:41.376 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.367+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:48:41.377 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.367+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:48:41.377 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.367+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:48:41.378 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.367+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:48:41.378 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.367+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:48:41.381 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.371+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:48:41.382 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.371+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:48:41.382 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.371+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:48:41.382 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.371+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:48:41.382 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.371+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:48:41.382 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.371+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:48:41.383 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:41.371+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:48:41.700 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:48:41.700 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:41.700 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:41.704 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:42.567 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:48:42.567 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:48:42.568 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863722 2023-10-21T04:48:42.568 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:42.568 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:48:42.568 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:48:42.568 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:48:47.314 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:47.307+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:48:47.318 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:48:47.311+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:48:52.577 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:48:52.580 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:48:52.581 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:48:52.590 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:48:53.473 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:48:53.474 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:48:53.477 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863733 2023-10-21T04:48:53.477 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:48:53.477 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:48:53.478 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:48:53.478 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:49:03.488 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:49:03.489 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:03.490 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:03.495 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:04.338 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:49:04.338 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:49:04.338 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863744 2023-10-21T04:49:04.339 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:04.339 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:49:04.339 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:49:04.339 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:49:14.340 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:49:14.341 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:14.341 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:14.346 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:15.259 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:49:15.270 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:49:15.281 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863755 2023-10-21T04:49:15.282 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:15.282 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:49:15.282 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:49:15.282 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:49:16.357 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:16.343+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:49:16.359 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:16.347+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:49:16.359 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:16.347+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:49:16.359 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:16.347+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:49:16.359 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:16.347+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:49:16.365 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:16.355+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:49:16.365 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:16.355+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:49:16.367 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:16.359+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:49:16.367 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:16.359+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:49:16.376 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:16.367+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:49:16.377 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:16.367+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:49:16.377 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:16.367+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:49:25.290 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:25.293 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:49:25.306 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:25.309 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:26.231 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:49:26.231 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:49:26.232 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863766 2023-10-21T04:49:26.232 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:26.232 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:49:26.233 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:49:26.233 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:49:36.239 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:49:36.240 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:36.245 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:36.266 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:37.157 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:49:37.158 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:49:37.158 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863777 2023-10-21T04:49:37.158 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:37.159 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:49:37.159 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:49:37.159 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:49:37.896 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:37.887+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:49:37.912 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:37.903+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:49:37.931 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:37.923+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:49:39.597 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:49:39.597 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:49:39.597 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:49:39.598 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:49:39.598 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:49:39.598 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:49:39.598 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:49:39.598 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:49:39.598 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:49:39.599 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:49:39.599 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:49:39.599 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:49:39.599 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:49:39.599 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:39.587+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:49:45.076 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:45.067+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:49:45.079 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:45.071+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:49:45.079 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:45.071+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:49:45.086 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:45.079+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:49:45.086 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:49:45.079+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:49:47.161 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:47.161 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:47.162 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:47.169 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:49:48.067 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:49:48.068 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:49:48.068 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863788 2023-10-21T04:49:48.068 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:48.069 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:49:48.069 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:49:48.069 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:49:58.072 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:49:58.072 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:49:58.073 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:49:58.083 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:49:58.983 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:49:58.984 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:49:58.984 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863798 2023-10-21T04:49:58.985 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:49:58.985 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:49:58.985 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:49:58.985 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:50:08.997 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:08.998 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:50:09.004 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:09.020 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:09.962 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:50:09.962 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:50:09.963 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863809 2023-10-21T04:50:09.964 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:09.964 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:50:09.964 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:50:09.964 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:50:19.969 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:19.969 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:19.972 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:19.976 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:50:21.057 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:50:21.058 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:50:21.059 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863821 2023-10-21T04:50:21.059 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:21.059 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:50:21.059 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:50:21.060 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:50:31.064 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:50:31.065 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:31.067 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:31.068 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:31.921 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:50:31.921 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:50:31.922 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863831 2023-10-21T04:50:31.922 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:31.922 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:50:31.922 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:50:31.923 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:50:41.923 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:50:41.924 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:41.924 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:41.930 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:42.907 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:50:42.907 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:50:42.908 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863842 2023-10-21T04:50:42.908 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:42.908 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:50:42.908 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:50:42.909 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:50:52.910 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:50:52.911 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:50:52.914 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:50:52.921 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:50:53.365 INFO:tasks.workunit.client.0.smithi133.stderr:2023-10-21T04:50:53.356+0000 7fdfca7fc640 0 --2- 172.21.15.133:0/2638031244 >> v2:172.21.15.133:3300/0 conn(0x7fdfcc12a310 0x7fdfcc1298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-21T04:50:54.206 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:50:54.207 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:50:54.207 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863854 2023-10-21T04:50:54.208 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:50:54.208 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:50:54.208 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:50:54.208 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:51:00.972 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:00.956+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:51:00.973 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:00.956+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:51:04.209 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:04.209 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:04.219 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:51:04.219 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:05.199 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:51:05.201 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:51:05.202 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863865 2023-10-21T04:51:05.202 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:05.203 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:51:05.203 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:51:05.203 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:51:12.848 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:12.836+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:51:12.869 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:12.848+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:51:12.870 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:12.848+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:51:15.217 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:15.230 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:15.231 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:15.231 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:51:16.108 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:51:16.109 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:51:16.109 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863876 2023-10-21T04:51:16.110 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:16.110 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:51:16.110 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:51:16.110 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:51:26.111 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:51:26.112 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:26.119 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:26.125 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:27.091 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:51:27.091 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:51:27.093 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863887 2023-10-21T04:51:27.094 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:27.094 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:51:27.094 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:51:27.094 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:51:37.099 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:51:37.104 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:37.110 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:37.112 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:38.125 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:51:38.125 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:51:38.126 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863898 2023-10-21T04:51:38.126 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:38.126 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:51:38.126 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:51:38.127 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:51:38.974 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:38.968+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:51:38.974 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:38.968+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:51:38.974 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:38.968+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:51:39.234 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:39.220+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:51:39.234 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:39.220+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:51:39.236 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:39.228+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:51:39.236 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:39.228+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:51:48.140 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:51:48.140 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:48.141 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:48.145 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:49.013 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:51:49.014 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:51:49.014 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863909 2023-10-21T04:51:49.014 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:49.015 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:51:49.015 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:51:49.015 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:51:51.389 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:51.368+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:51:51.389 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:51.376+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:51:51.389 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:51.376+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:51:51.392 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:51:51.384+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:51:59.017 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:51:59.018 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:51:59.019 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:51:59.021 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:51:59.891 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:51:59.892 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:51:59.892 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863919 2023-10-21T04:51:59.892 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:51:59.893 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:51:59.893 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:51:59.893 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:52:04.339 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:04.324+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:52:04.341 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:04.332+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:52:04.341 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:04.332+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:52:04.350 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:04.344+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:52:04.351 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:04.344+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:52:09.897 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:09.898 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:52:09.900 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:09.903 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:10.268 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:10.260+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:52:10.269 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:10.260+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:52:10.269 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:10.260+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:52:10.269 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:10.260+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:52:10.277 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:10.268+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:52:10.293 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:10.284+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:52:10.293 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:10.284+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:52:10.295 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:10.288+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:52:10.296 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:10.288+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:52:10.296 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:10.288+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:52:10.921 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:52:10.921 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:52:10.922 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863930 2023-10-21T04:52:10.922 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:10.922 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:52:10.923 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:52:10.923 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:52:11.011 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.000+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:52:11.012 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:52:11.012 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:52:11.012 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:52:11.012 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:52:11.012 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:52:11.014 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.008+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:52:11.014 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.008+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:52:11.014 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.008+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:52:11.014 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.008+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:52:11.014 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.008+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:52:11.015 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.008+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:52:11.015 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:11.008+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:52:15.553 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:52:15.554 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:52:15.554 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:52:15.554 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:52:15.554 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:52:15.554 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:52:15.555 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:52:15.555 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:52:15.555 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:52:15.555 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:52:15.555 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:52:15.556 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:52:15.556 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:52:15.556 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:15.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:52:20.924 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:52:20.925 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:20.937 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:20.947 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:21.756 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:52:21.756 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:52:21.757 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863941 2023-10-21T04:52:21.757 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:21.757 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:52:21.758 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:52:21.758 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:52:31.765 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:52:31.780 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:31.783 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:31.793 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:32.823 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:52:32.825 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:52:32.826 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863952 2023-10-21T04:52:32.827 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:32.827 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:52:32.827 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:52:32.827 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:52:42.831 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:42.832 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:52:42.844 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:42.849 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:43.824 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:52:43.824 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:52:43.825 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863963 2023-10-21T04:52:43.825 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:43.826 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:52:43.826 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:52:43.826 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:52:53.830 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:52:53.830 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:52:53.831 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:52:53.833 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:52:54.655 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:52:54.656 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:52:54.656 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863974 2023-10-21T04:52:54.656 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:52:54.657 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:52:54.657 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:52:54.657 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:52:55.125 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:55.120+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:52:55.149 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:55.144+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:52:55.150 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:55.144+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:52:55.165 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:55.156+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:52:55.165 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:52:55.156+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:53:04.658 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:53:04.658 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:04.659 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:04.667 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:05.578 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:53:05.578 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:53:05.579 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863985 2023-10-21T04:53:05.579 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:05.579 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:53:05.580 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:53:05.580 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:53:15.594 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:15.602 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:53:15.605 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:15.614 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:16.600 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:53:16.601 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:53:16.601 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697863996 2023-10-21T04:53:16.602 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:16.602 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:53:16.602 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:53:16.602 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:53:26.614 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:26.617 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:26.619 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:26.621 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:53:27.593 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:53:27.593 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:53:27.594 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864007 2023-10-21T04:53:27.594 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:27.595 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:53:27.595 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:53:27.595 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:53:28.020 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:28.012+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:53:28.041 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:28.028+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:53:28.041 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:28.028+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:53:28.041 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:28.028+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:53:28.042 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:28.028+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:53:28.050 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:28.044+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:53:28.051 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:28.044+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:53:28.052 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:28.044+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:53:28.052 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:28.044+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:53:28.065 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:28.056+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:53:28.065 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:28.056+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:53:28.065 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:28.056+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:53:35.981 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:35.972+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:53:35.981 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:35.972+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:53:35.985 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:35.976+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:53:35.986 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:35.980+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:53:35.989 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:35.980+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:53:35.989 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:35.980+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:53:35.995 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:35.988+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:53:35.995 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:35.988+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:53:36.002 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:35.996+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:53:36.002 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:35.996+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:53:36.013 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:53:36.013 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:53:36.014 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:53:36.014 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:53:36.014 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:53:36.014 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:53:36.014 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:53:36.015 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:53:36.015 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:53:36.015 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:53:36.015 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:53:36.015 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:53:36.016 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:53:36.016 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:53:36.016 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:36.004+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:53:37.596 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:37.596 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:37.608 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:37.610 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:53:38.489 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:53:38.494 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:53:38.495 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864018 2023-10-21T04:53:38.496 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:38.496 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:53:38.496 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:53:38.496 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:53:47.862 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:47.852+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:53:47.868 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:47.860+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:53:47.869 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:53:47.860+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:53:48.511 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:48.512 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:53:48.512 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:48.520 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:53:49.470 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:53:49.470 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:53:49.471 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864029 2023-10-21T04:53:49.471 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:53:49.471 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:53:49.471 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:53:49.472 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:53:59.478 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:53:59.478 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:53:59.479 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:53:59.485 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:00.394 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:54:00.394 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:54:00.396 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864040 2023-10-21T04:54:00.396 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:00.396 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:54:00.397 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:54:00.397 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:54:01.133 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T04:54:01.134 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:54:01.134 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:54:01.134 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:54:01.134 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:54:01.134 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:54:01.135 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:54:01.135 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:54:01.135 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:54:01.135 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:54:01.135 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:54:01.136 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:54:01.136 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:54:01.136 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:01.128+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:54:10.410 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:10.410 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:10.410 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:54:10.415 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:11.284 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:54:11.285 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:54:11.285 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864051 2023-10-21T04:54:11.286 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:11.286 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:54:11.286 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:54:11.286 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:54:16.439 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:16.428+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:54:16.439 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:16.428+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:54:16.439 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:16.428+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:54:16.440 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:16.428+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:54:16.546 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:16.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:54:16.547 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:16.540+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:54:16.554 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:16.548+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:54:16.555 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:16.548+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:54:21.288 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:54:21.293 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:21.305 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:21.305 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:22.228 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:54:22.229 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:54:22.229 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864062 2023-10-21T04:54:22.229 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:22.230 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:54:22.230 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:54:22.230 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:54:32.243 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:54:32.244 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:32.244 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:32.249 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:33.142 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:54:33.142 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:54:33.143 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864073 2023-10-21T04:54:33.143 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:33.144 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:54:33.144 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:54:33.144 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:54:43.145 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:54:43.148 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:43.155 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:43.157 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:44.065 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:54:44.065 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:54:44.065 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864084 2023-10-21T04:54:44.066 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:44.066 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:54:44.066 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:54:44.066 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:54:44.081 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:44.072+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:54:44.085 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:44.076+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:54:44.085 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:44.076+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:54:44.085 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:44.076+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:54:44.086 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:44.080+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:54:44.092 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:44.080+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:54:44.096 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:44.088+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:54:44.097 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:44.088+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:54:44.105 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:44.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:54:44.105 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:44.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:54:44.105 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:54:44.092+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:54:54.070 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:54:54.080 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:54:54.081 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:54:54.082 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:54:54.953 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:54:54.954 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:54:54.954 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864094 2023-10-21T04:54:54.955 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:54:54.955 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:54:54.955 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:54:54.955 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:55:04.956 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:55:04.956 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:04.972 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:04.978 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:05.750 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:55:05.750 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:55:05.751 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864105 2023-10-21T04:55:05.751 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:05.751 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:55:05.751 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:55:05.752 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:55:15.753 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:55:15.755 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:15.757 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:15.769 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:16.659 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:55:16.659 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:55:16.660 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864116 2023-10-21T04:55:16.661 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:16.661 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:55:16.661 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:55:16.661 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:55:26.669 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:55:26.674 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:26.674 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:26.677 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:27.548 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:55:27.548 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:55:27.549 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864127 2023-10-21T04:55:27.549 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:27.549 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:55:27.550 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:55:27.550 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:55:36.965 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:36.947+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:55:36.967 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:36.959+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:55:36.967 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:36.959+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:55:36.967 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:36.959+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:55:36.968 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:36.959+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:55:36.968 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:36.959+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:55:36.968 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:36.959+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:55:36.969 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:36.959+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:55:36.969 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:36.959+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:55:37.554 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:37.555 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:55:37.567 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:37.578 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:38.474 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:55:38.477 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:55:38.478 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864138 2023-10-21T04:55:38.478 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:38.478 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:55:38.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:55:38.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:55:43.498 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:43.479+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:55:43.499 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:43.483+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:55:43.499 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:43.487+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:55:43.499 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:43.487+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:55:43.499 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:43.487+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:55:43.500 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:43.491+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:55:43.500 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:43.491+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:55:43.522 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:43.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:55:43.522 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:43.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:55:43.522 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:43.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:55:43.522 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:43.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:55:43.523 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:55:43.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:55:48.482 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:48.484 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:48.485 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:55:48.497 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:55:49.458 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:55:49.461 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:55:49.462 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864149 2023-10-21T04:55:49.462 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:55:49.462 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:55:49.463 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:55:49.463 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:55:59.464 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:55:59.464 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:55:59.464 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:55:59.465 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:00.408 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:56:00.410 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:56:00.413 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864160 2023-10-21T04:56:00.413 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:00.414 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:56:00.414 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:56:00.414 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:56:10.419 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:10.420 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:10.433 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:10.446 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:56:11.309 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:56:11.314 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:56:11.315 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864171 2023-10-21T04:56:11.316 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:11.316 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:56:11.316 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:56:11.316 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:56:21.320 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:21.327 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:21.327 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:21.329 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:56:22.293 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:56:22.294 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:56:22.295 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864182 2023-10-21T04:56:22.295 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:22.295 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:56:22.296 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:56:22.296 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:56:31.607 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:31.599+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:56:31.626 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:31.619+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:56:31.627 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:31.619+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:56:31.627 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:31.619+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:56:31.627 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:31.619+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:56:31.627 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:31.619+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:56:31.632 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:31.623+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:56:31.632 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:31.623+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:56:31.633 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:31.623+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:56:31.633 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:31.623+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:56:31.633 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:31.623+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:56:31.633 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:31.623+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:56:32.297 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:32.297 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:32.297 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:32.301 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:56:33.249 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:56:33.249 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:56:33.250 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864193 2023-10-21T04:56:33.250 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:33.250 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:56:33.251 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:56:33.251 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:56:43.252 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:43.254 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:56:43.254 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:43.255 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:44.070 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:56:44.071 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:56:44.071 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864204 2023-10-21T04:56:44.071 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:44.072 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:56:44.072 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:56:44.072 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:56:54.073 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:56:54.073 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:56:54.074 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:56:54.074 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:56:55.066 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:56:55.068 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:56:55.077 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864215 2023-10-21T04:56:55.078 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:56:55.078 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:56:55.078 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:56:55.079 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:56:56.579 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:56.559+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:56:56.589 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:56:56.563+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:57:05.093 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:05.094 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:05.098 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:05.113 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:57:06.100 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:57:06.101 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:57:06.101 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864226 2023-10-21T04:57:06.101 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:06.102 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:57:06.102 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:57:06.102 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:57:16.104 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:57:16.105 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:16.109 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:16.125 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:17.044 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:57:17.045 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:57:17.046 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864237 2023-10-21T04:57:17.047 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:17.047 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:57:17.047 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:57:17.047 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:57:27.051 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:57:27.052 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:27.053 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:27.060 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:27.992 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:57:27.994 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:57:27.995 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864247 2023-10-21T04:57:27.995 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:27.995 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:57:27.996 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:57:27.996 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:57:37.998 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:57:38.003 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:38.004 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:38.004 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:38.838 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:57:38.838 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:57:38.839 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864258 2023-10-21T04:57:38.839 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:38.839 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:57:38.839 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:57:38.840 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:57:39.380 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:39.367+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:57:39.381 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:39.371+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:57:39.391 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:39.383+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:57:39.395 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:39.387+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:57:39.399 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:39.391+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:57:39.400 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:39.391+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:57:45.720 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.711+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:57:45.731 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.723+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:57:45.731 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.723+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:57:45.732 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.723+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:57:45.732 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.723+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:57:45.732 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.723+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:57:45.750 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.743+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:57:45.750 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.743+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:57:45.750 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.743+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:57:45.751 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.743+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:57:45.751 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.743+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:57:45.751 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.743+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:57:45.752 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.743+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:57:45.752 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.743+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:57:45.752 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.743+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:57:45.752 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:45.743+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:57:48.846 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:57:48.847 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:48.848 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:57:48.856 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:49.696 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:57:49.697 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:57:49.698 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864269 2023-10-21T04:57:49.699 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:57:49.699 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:57:49.699 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:57:49.699 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:57:59.216 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:59.203+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:57:59.217 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:59.203+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:57:59.217 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:59.207+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:57:59.218 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:57:59.207+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:57:59.708 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:57:59.708 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:57:59.715 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:57:59.728 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:00.574 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:58:00.574 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:58:00.575 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864280 2023-10-21T04:58:00.575 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:00.575 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:58:00.576 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:58:00.576 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:58:10.583 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:10.583 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:10.583 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:10.593 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:58:11.576 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:58:11.577 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:58:11.577 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864291 2023-10-21T04:58:11.577 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:11.578 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:58:11.578 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:58:11.578 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:58:21.579 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:58:21.580 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:21.581 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:21.605 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:22.493 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:58:22.494 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:58:22.495 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864302 2023-10-21T04:58:22.496 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:22.496 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:58:22.496 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:58:22.496 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:58:32.497 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:58:32.498 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:32.498 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:32.499 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:33.352 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:58:33.352 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:58:33.353 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864313 2023-10-21T04:58:33.353 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:33.353 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:58:33.353 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:58:33.354 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:58:43.354 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:58:43.355 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:43.358 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:43.368 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:44.332 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:58:44.332 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:58:44.333 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864324 2023-10-21T04:58:44.333 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:44.334 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:58:44.334 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:58:44.334 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:58:54.342 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:58:54.343 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:58:54.350 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:58:54.371 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:58:55.341 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:58:55.341 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:58:55.342 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864335 2023-10-21T04:58:55.342 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:58:55.342 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:58:55.342 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:58:55.343 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:58:56.773 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:58:56.763+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:58:56.774 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:58:56.767+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:59:05.344 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:05.345 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:05.349 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:05.372 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:59:06.334 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:59:06.335 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:59:06.336 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864346 2023-10-21T04:59:06.336 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:06.336 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:59:06.336 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:59:06.336 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:59:16.337 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:16.338 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:16.339 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:59:16.341 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:17.296 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:59:17.299 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:59:17.301 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864357 2023-10-21T04:59:17.301 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:17.301 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:59:17.302 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:59:17.302 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:59:22.775 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:22.755+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:59:22.776 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:22.755+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:59:22.776 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:22.755+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:59:22.776 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:22.755+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:59:22.776 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:22.755+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:59:22.776 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:22.755+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:59:22.777 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:22.755+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:59:22.777 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:22.755+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:59:22.777 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:22.755+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:59:22.777 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:22.755+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:59:22.777 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:22.755+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:59:22.778 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:22.755+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:59:27.308 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:59:27.308 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:27.319 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:27.321 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:28.277 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:59:28.278 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:59:28.278 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864368 2023-10-21T04:59:28.279 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:28.279 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:59:28.279 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:59:28.279 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:59:29.167 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.159+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:59:29.173 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.163+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:59:29.174 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.163+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:59:29.174 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.163+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:59:29.183 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.175+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:59:29.184 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.175+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:59:29.188 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.175+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:59:29.188 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.175+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:59:29.190 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.183+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:59:29.190 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.183+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:59:29.190 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.183+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:59:29.190 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.183+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:59:29.191 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:29.183+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:59:30.297 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:30.291+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:59:30.299 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:30.291+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:59:30.300 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:30.291+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:59:30.300 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:30.291+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:59:30.315 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:30.303+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:59:30.318 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:30.311+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:59:30.319 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:30.311+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:59:30.320 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:30.311+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:59:30.321 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:30.311+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:59:30.323 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:30.315+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:59:30.324 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:30.315+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:59:30.324 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:30.315+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:59:38.281 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:59:38.286 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:38.289 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:38.290 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:39.228 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:59:39.229 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:59:39.230 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864379 2023-10-21T04:59:39.230 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:39.230 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:59:39.230 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:59:39.231 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T04:59:40.921 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:40.907+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:59:40.922 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:40.911+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:59:41.414 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.403+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T04:59:41.428 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.419+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T04:59:41.428 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.419+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T04:59:41.429 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.419+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T04:59:41.429 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.419+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T04:59:41.429 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.419+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T04:59:41.434 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.427+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T04:59:41.434 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.427+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T04:59:41.434 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.427+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T04:59:41.434 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.427+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T04:59:41.435 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.427+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T04:59:41.435 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.427+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T04:59:41.435 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T04:59:41.427+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T04:59:49.235 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T04:59:49.238 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T04:59:49.239 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T04:59:49.249 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T04:59:50.162 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T04:59:50.162 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T04:59:50.163 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864390 2023-10-21T04:59:50.163 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T04:59:50.163 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T04:59:50.164 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T04:59:50.164 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:00:00.176 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:00.185 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:00:00.185 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:00.186 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:01.130 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:00:01.131 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:00:01.131 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864401 2023-10-21T05:00:01.132 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:01.132 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:00:01.132 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:00:01.132 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:00:05.490 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:05.475+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:00:05.490 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:05.475+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:00:05.490 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:05.475+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:00:05.494 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:05.483+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:00:11.133 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:00:11.133 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:11.138 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:11.140 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:12.041 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:00:12.042 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:00:12.042 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864412 2023-10-21T05:00:12.043 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:12.043 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:00:12.043 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:00:12.043 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:00:22.044 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:00:22.045 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:22.045 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:22.049 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:22.657 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:22.647+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:00:22.662 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:22.655+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:00:22.666 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:22.659+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:00:22.690 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:22.683+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:00:22.691 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:22.683+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:00:22.905 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:00:22.905 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:00:22.906 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864422 2023-10-21T05:00:22.906 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:22.906 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:00:22.906 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:00:22.907 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:00:32.908 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:32.908 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:32.909 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:00:32.913 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:33.874 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:00:33.879 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:00:33.881 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864433 2023-10-21T05:00:33.881 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:33.881 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:00:33.881 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:00:33.881 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:00:39.950 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:39.943+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:00:39.971 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:39.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:00:43.310 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:43.303+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:00:43.312 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:43.303+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:00:43.312 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:43.303+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:00:43.312 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:43.303+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:00:43.312 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:43.303+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:00:43.326 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:43.319+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:00:43.339 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:43.327+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:00:43.339 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:43.327+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:00:43.339 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:43.327+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:00:43.340 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:43.327+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:00:43.340 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:43.327+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:00:43.340 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:43.327+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:00:43.892 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:43.892 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:00:43.893 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:43.900 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:44.813 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:00:44.814 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:00:44.815 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864444 2023-10-21T05:00:44.815 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:44.815 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:00:44.815 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:00:44.815 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:00:54.826 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:00:54.837 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:00:54.838 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:00:54.843 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:00:55.702 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:00:55.702 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:00:55.703 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864455 2023-10-21T05:00:55.703 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:00:55.703 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:00:55.704 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:00:55.704 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:00:56.339 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:56.331+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:00:56.366 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:56.355+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:00:56.366 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:00:56.355+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:01:05.705 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:01:05.705 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:05.706 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:05.710 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:06.694 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:01:06.694 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:01:06.695 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864466 2023-10-21T05:01:06.695 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:06.695 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:01:06.696 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:01:06.696 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:01:13.938 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:01:13.931+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:01:13.949 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:01:13.939+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:01:13.950 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:01:13.939+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:01:13.951 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:01:13.943+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:01:13.952 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:01:13.943+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:01:16.696 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:01:16.697 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:16.703 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:16.718 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:17.587 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:01:17.588 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:01:17.588 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864477 2023-10-21T05:01:17.588 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:17.589 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:01:17.589 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:01:17.589 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:01:27.590 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:27.590 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:01:27.591 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:27.611 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:28.500 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:01:28.500 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:01:28.501 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864488 2023-10-21T05:01:28.501 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:28.501 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:01:28.502 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:01:28.502 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:01:38.505 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:38.509 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:38.511 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:01:38.514 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:39.476 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:01:39.476 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:01:39.477 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864499 2023-10-21T05:01:39.477 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:39.478 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:01:39.478 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:01:39.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:01:49.490 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:01:49.491 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:01:49.491 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:01:49.493 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:01:50.368 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:01:50.368 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:01:50.369 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864510 2023-10-21T05:01:50.369 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:01:50.370 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:01:50.370 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:01:50.370 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:02:00.371 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:00.372 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:00.372 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:00.388 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:02:01.358 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:02:01.370 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:02:01.373 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864521 2023-10-21T05:02:01.373 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:01.373 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:02:01.373 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:02:01.373 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:02:11.383 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:02:11.384 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:11.384 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:11.391 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:11.786 INFO:tasks.workunit.client.0.smithi133.stderr:2023-10-21T05:02:11.783+0000 7f889259c640 0 --2- 172.21.15.133:0/2120722144 >> v2:172.21.15.133:3300/0 conn(0x7f888c0202f0 0x7f888c0206d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-21T05:02:12.455 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:02:12.457 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:02:12.458 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864532 2023-10-21T05:02:12.459 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:12.459 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:02:12.459 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:02:12.459 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:02:12.709 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:12.707+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:02:12.722 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:12.707+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T05:02:12.722 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:12.707+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:02:12.722 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:12.711+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:02:12.727 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:12.723+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:02:12.728 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:12.723+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:02:12.728 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:12.723+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:02:12.728 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:12.723+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:02:12.728 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:12.723+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:02:22.465 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:02:22.465 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:22.466 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:22.472 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:23.365 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:02:23.369 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:02:23.371 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864543 2023-10-21T05:02:23.371 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:23.372 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:02:23.372 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:02:23.372 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:02:28.784 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:28.775+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:02:28.784 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:28.779+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:02:28.784 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:28.779+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:02:28.796 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:28.783+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:02:28.813 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:28.795+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:02:28.813 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:28.795+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:02:33.373 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:02:33.375 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:33.375 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:33.386 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:34.358 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:02:34.359 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:02:34.359 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864554 2023-10-21T05:02:34.360 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:34.360 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:02:34.360 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:02:34.360 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:02:44.367 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:44.373 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:44.373 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:02:44.378 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:45.316 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:02:45.316 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:02:45.317 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864565 2023-10-21T05:02:45.317 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:45.317 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:02:45.317 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:02:45.318 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:02:55.319 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:02:55.319 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:02:55.320 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:02:55.335 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:02:56.268 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:02:56.273 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:02:56.274 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864576 2023-10-21T05:02:56.274 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:02:56.275 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:02:56.275 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:02:56.275 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:02:59.679 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.671+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:02:59.682 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.679+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:02:59.682 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.679+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:02:59.682 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.679+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:02:59.682 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.679+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T05:02:59.683 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.679+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:02:59.694 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.687+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:02:59.694 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.687+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:02:59.694 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.687+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:02:59.695 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.687+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:02:59.695 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.687+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:02:59.695 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.687+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:02:59.695 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:02:59.687+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:03:06.277 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:06.278 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:06.278 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:03:06.290 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:07.201 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:03:07.202 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:03:07.202 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864587 2023-10-21T05:03:07.203 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:07.203 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ -n 172.21.15.133:0/3061939021 ]] 2023-10-21T05:03:07.203 INFO:tasks.workunit.client.0.smithi133.stderr:+ [[ 172.21.15.133:0/3061939021 != \1\7\2\.\2\1\.\1\5\.\1\3\3\:\0\/\3\0\6\1\9\3\9\0\2\1 ]] 2023-10-21T05:03:07.203 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:03:17.204 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq '.active_clients[]' 2023-10-21T05:03:17.208 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-21T05:03:17.210 INFO:tasks.workunit.client.0.smithi133.stderr:++ ceph mgr dump 2023-10-21T05:03:17.228 INFO:tasks.workunit.client.0.smithi133.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-21T05:03:18.065 INFO:tasks.workunit.client.0.smithi133.stderr:+ CLIENT_ADDR=172.21.15.133:0/3061939021 2023-10-21T05:03:18.066 INFO:tasks.workunit.client.0.smithi133.stderr:++ date +%s 2023-10-21T05:03:18.067 INFO:tasks.workunit.client.0.smithi133.stderr:+ CURRENT_TIME=1697864598 2023-10-21T05:03:18.067 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-21T05:03:18.067 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i = 1 )) 2023-10-21T05:03:18.068 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:03:18.068 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:03:18.335 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:03:18.331+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:03:18.336 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:03:18.354 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:03:28.361 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:03:28.362 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:03:28.362 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:03:28.454 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:03:28.451+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:03:28.455 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:03:28.466 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:03:38.473 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:03:38.474 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:03:38.474 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:03:38.536 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:03:38.531+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:03:38.537 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:03:38.542 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:03:48.550 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:03:48.550 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:03:48.551 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:03:48.636 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:03:48.631+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:03:48.636 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:03:48.650 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:03:58.652 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:03:58.653 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:03:58.653 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:03:58.729 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:03:58.723+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:03:58.730 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:03:58.737 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:04:04.121 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:04:04.115+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:04:04.125 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:04:04.119+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:04:04.125 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:04:04.119+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:04:04.126 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:04:04.119+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:04:04.126 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:04:04.119+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:04:04.126 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:04:04.119+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:04:04.126 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:04:04.119+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:04:04.127 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:04:04.119+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:04:08.739 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:04:08.739 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:04:08.739 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:04:08.825 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:04:08.823+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:04:08.826 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:04:08.836 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:04:18.843 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:04:18.843 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:04:18.843 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:04:18.935 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:04:18.931+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:04:18.936 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:04:18.945 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:04:28.946 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:04:28.946 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:04:28.946 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:04:29.046 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:04:29.043+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:04:29.047 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:04:29.051 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:04:29.183 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:04:29.167+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:04:29.185 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:04:29.167+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:04:29.185 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:04:29.167+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:04:39.053 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:04:39.053 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:04:39.053 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:04:39.136 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:04:39.131+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:04:39.137 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:04:39.153 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:04:49.159 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:04:49.160 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:04:49.160 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:04:49.258 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:04:49.255+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:04:49.260 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:04:49.272 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:04:59.285 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:04:59.285 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:04:59.285 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:04:59.371 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:04:59.367+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:04:59.371 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:04:59.381 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:05:09.383 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:05:09.383 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:05:09.384 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:05:09.462 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:05:09.455+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:05:09.462 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:05:09.469 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:05:19.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:05:19.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:05:19.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:05:19.583 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:05:19.579+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:05:19.583 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:05:19.590 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:05:29.595 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:05:29.595 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:05:29.595 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:05:29.681 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:05:29.675+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:05:29.682 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:05:29.698 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:05:30.090 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:30.079+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:05:30.091 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:30.087+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:05:30.091 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:30.087+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:05:30.091 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:30.087+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:05:30.092 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:30.087+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:05:30.093 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:30.087+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:05:30.102 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:30.095+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:05:30.103 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:30.095+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:05:30.104 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:30.099+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:05:30.105 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:30.099+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:05:30.105 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:30.099+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:05:39.699 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:05:39.700 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:05:39.700 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:05:39.803 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:05:39.799+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:05:39.804 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:05:39.810 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:05:49.811 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:05:49.812 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:05:49.812 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:05:49.885 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:05:49.879+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:05:49.889 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:05:49.907 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:05:51.443 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:51.439+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:05:51.466 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:51.463+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:05:51.466 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:51.463+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:05:51.466 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:51.463+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:05:51.467 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:51.463+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:05:59.274 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:59.271+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:05:59.275 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:59.271+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:05:59.275 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:59.271+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:05:59.275 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:59.271+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:05:59.298 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:59.295+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:05:59.299 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:59.295+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:05:59.299 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:59.295+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:05:59.316 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:59.311+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:05:59.316 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:59.311+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:05:59.316 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:05:59.311+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:05:59.912 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:05:59.912 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:05:59.913 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:06:00.008 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:06:00.008 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:06:00.003+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:06:00.013 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:06:10.014 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:06:10.014 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:06:10.015 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:06:10.083 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:06:10.079+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:06:10.083 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:06:10.093 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:06:20.094 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:06:20.095 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:06:20.095 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:06:20.162 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:06:20.159+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:06:20.163 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:06:20.170 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:06:30.172 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:06:30.172 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:06:30.173 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:06:30.264 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:06:30.259+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:06:30.265 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:06:30.275 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:06:38.368 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:38.359+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:06:38.368 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:38.359+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:06:38.378 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:38.371+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:06:38.402 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:38.395+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:06:38.402 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:38.395+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:06:38.403 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:38.395+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:06:38.403 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:38.395+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:06:38.403 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:38.395+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:06:40.276 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:06:40.276 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:06:40.276 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:06:40.352 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:06:40.347+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:06:40.353 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:06:40.363 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:06:50.365 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:06:50.365 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:06:50.366 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:06:50.465 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:06:50.459+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:06:50.466 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:06:50.478 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:06:55.879 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:55.875+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:06:55.879 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:55.875+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:06:55.879 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:55.875+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:06:55.886 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:55.883+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:06:55.910 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:55.907+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:06:55.911 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:06:55.907+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:07:00.479 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:07:00.480 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:07:00.480 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:00.552 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:07:00.547+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:00.553 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:00.557 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:07:02.932 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.927+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:07:02.946 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.939+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:07:02.946 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.939+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:07:02.948 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.943+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:07:02.948 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.943+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T05:07:02.948 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.943+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:07:02.957 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:07:02.957 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:07:02.957 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:07:02.957 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:07:02.958 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:07:02.958 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:07:02.958 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:07:02.958 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:07:02.958 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:07:02.959 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:07:02.959 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:07:02.959 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:07:02.959 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:07:02.959 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:07:02.960 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:07:02.960 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:07:02.951+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:07:10.560 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:07:10.561 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:07:10.561 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-21T05:07:10.664 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:07:10.659+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:10.665 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:10.680 INFO:tasks.workunit.client.0.smithi133.stderr:+ sleep 10 2023-10-21T05:07:20.682 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i++ )) 2023-10-21T05:07:20.683 INFO:tasks.workunit.client.0.smithi133.stderr:+ (( i <= 24 )) 2023-10-21T05:07:20.683 INFO:tasks.workunit.client.0.smithi133.stderr:+ grep 'every 2m' 2023-10-21T05:07:20.689 INFO:tasks.workunit.client.0.smithi133.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-10-21T05:07:20.774 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:07:20.771+0000 7fd1f061e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-21T05:07:20.775 INFO:tasks.workunit.client.0.smithi133.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-21T05:07:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T05:07:20.787 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-21T05:07:20.787 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-21T05:07:21.446 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-21T05:07:21.507 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3dd87c3216304c02862d96ba3bd2d91b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-21T05:07:21.510 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-21T05:07:21.581 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-21T05:07:21.589 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-21T05:07:21.589 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:07:22.178 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:07:22.179 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:07:22.207 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2059,"stamp":"2023-10-21T05:07:22.075786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26530086017,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1748207,"num_read_kb":6814061,"num_write":1725895,"num_write_kb":6902823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76188,"ondisk_log_size":76188,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18236764,"kb_used_data":13804612,"kb_used_omap":1351,"kb_used_meta":4430776,"kb_avail":264878756,"statfs":{"total":289910292480,"available":271235846144,"internally_reserved":0,"allocated":14135922688,"data_stored":17029259706,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1383519,"internal_metadata":4537115553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,4,3,14,7,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18190336,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4765,"num_read_kb":19044,"num_write":5215,"num_write_kb":20860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006957"},"pg_stats":[{"pgid":"2.7","version":"84'12096","reported_seq":16645,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.319897+0000","last_change":"2023-10-21T04:01:35.655104+0000","last_active":"2023-10-21T05:07:20.319897+0000","last_peered":"2023-10-21T05:07:20.319897+0000","last_clean":"2023-10-21T05:07:20.319897+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:07:20.319897+0000","last_undegraded":"2023-10-21T05:07:20.319897+0000","last_fullsized":"2023-10-21T05:07:20.319897+0000","mapping_epoch":13,"log_start":"59'2000","ondisk_log_start":"59'2000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:01:13.743335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"84'6171","reported_seq":9475,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.292008+0000","last_change":"2023-10-21T04:01:35.652553+0000","last_active":"2023-10-21T05:07:20.292008+0000","last_peered":"2023-10-21T05:07:20.292008+0000","last_clean":"2023-10-21T05:07:20.292008+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:07:20.292008+0000","last_undegraded":"2023-10-21T05:07:20.292008+0000","last_fullsized":"2023-10-21T05:07:20.292008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:04:21.394002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"84'3083","reported_seq":4365,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.094379+0000","last_change":"2023-10-21T04:01:35.651529+0000","last_active":"2023-10-21T05:07:20.094379+0000","last_peered":"2023-10-21T05:07:20.094379+0000","last_clean":"2023-10-21T05:07:20.094379+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:07:20.094379+0000","last_undegraded":"2023-10-21T05:07:20.094379+0000","last_fullsized":"2023-10-21T05:07:20.094379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:33:14.743990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"84'6115","reported_seq":8829,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.295321+0000","last_change":"2023-10-21T04:01:35.652367+0000","last_active":"2023-10-21T05:07:20.295321+0000","last_peered":"2023-10-21T05:07:20.295321+0000","last_clean":"2023-10-21T05:07:20.295321+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:07:20.295321+0000","last_undegraded":"2023-10-21T05:07:20.295321+0000","last_fullsized":"2023-10-21T05:07:20.295321+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":6115,"log_dups_size":0,"ondisk_log_size":6115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:07:52.675494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"84'439867","reported_seq":876521,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:18.947939+0000","last_change":"2023-10-21T04:14:04.720485+0000","last_active":"2023-10-21T05:07:18.947939+0000","last_peered":"2023-10-21T05:07:18.947939+0000","last_clean":"2023-10-21T05:07:18.947939+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:07:18.947939+0000","last_undegraded":"2023-10-21T05:07:18.947939+0000","last_fullsized":"2023-10-21T05:07:18.947939+0000","mapping_epoch":16,"log_start":"82'429814","ondisk_log_start":"82'429814","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:20:02.233184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6720344064,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":438140,"num_read_kb":1722828,"num_write":438195,"num_write_kb":1752780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"62'27","reported_seq":191,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:06:33.659035+0000","last_change":"2023-10-21T04:01:35.650136+0000","last_active":"2023-10-21T05:06:33.659035+0000","last_peered":"2023-10-21T05:06:33.659035+0000","last_clean":"2023-10-21T05:06:33.659035+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:06:33.659035+0000","last_undegraded":"2023-10-21T05:06:33.659035+0000","last_fullsized":"2023-10-21T05:06:33.659035+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:55:55.905920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"84'443709","reported_seq":885719,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.729509+0000","last_change":"2023-10-21T04:14:04.706949+0000","last_active":"2023-10-21T05:07:20.729509+0000","last_peered":"2023-10-21T05:07:20.729509+0000","last_clean":"2023-10-21T05:07:20.729509+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:07:20.729509+0000","last_undegraded":"2023-10-21T05:07:20.729509+0000","last_fullsized":"2023-10-21T05:07:20.729509+0000","mapping_epoch":16,"log_start":"82'433618","ondisk_log_start":"82'433618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:21:43.744488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6830796800,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":443575,"num_read_kb":1743896,"num_write":441967,"num_write_kb":1767868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"80'133","reported_seq":3544,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:16.124984+0000","last_change":"2023-10-21T04:01:35.652988+0000","last_active":"2023-10-21T05:07:16.124984+0000","last_peered":"2023-10-21T05:07:16.124984+0000","last_clean":"2023-10-21T05:07:16.124984+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:07:16.124984+0000","last_undegraded":"2023-10-21T05:07:16.124984+0000","last_fullsized":"2023-10-21T05:07:16.124984+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:39:12.668814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"84'415012","reported_seq":829475,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:18.924162+0000","last_change":"2023-10-21T04:14:04.721281+0000","last_active":"2023-10-21T05:07:18.924162+0000","last_peered":"2023-10-21T05:07:18.924162+0000","last_clean":"2023-10-21T05:07:18.924162+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:07:18.924162+0000","last_undegraded":"2023-10-21T05:07:18.924162+0000","last_fullsized":"2023-10-21T05:07:18.924162+0000","mapping_epoch":16,"log_start":"82'404918","ondisk_log_start":"82'404918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:54:59.670291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6332669952,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":415827,"num_read_kb":1634656,"num_write":413461,"num_write_kb":1653844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"80'130","reported_seq":3475,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:15.916316+0000","last_change":"2023-10-21T04:01:35.650390+0000","last_active":"2023-10-21T05:07:15.916316+0000","last_peered":"2023-10-21T05:07:15.916316+0000","last_clean":"2023-10-21T05:07:15.916316+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:07:15.916316+0000","last_undegraded":"2023-10-21T05:07:15.916316+0000","last_fullsized":"2023-10-21T05:07:15.916316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:08:09.428828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"84'433225","reported_seq":863299,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.751989+0000","last_change":"2023-10-21T04:14:04.713203+0000","last_active":"2023-10-21T05:07:20.751989+0000","last_peered":"2023-10-21T05:07:20.751989+0000","last_clean":"2023-10-21T05:07:20.751989+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:07:20.751989+0000","last_undegraded":"2023-10-21T05:07:20.751989+0000","last_fullsized":"2023-10-21T05:07:20.751989+0000","mapping_epoch":16,"log_start":"82'423224","ondisk_log_start":"82'423224","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:49:18.294604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6645682195,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":431577,"num_read_kb":1696360,"num_write":431534,"num_write_kb":1726130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"84'15097","reported_seq":20883,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.351624+0000","last_change":"2023-10-21T04:01:35.650183+0000","last_active":"2023-10-21T05:07:20.351624+0000","last_peered":"2023-10-21T05:07:20.351624+0000","last_clean":"2023-10-21T05:07:20.351624+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:07:20.351624+0000","last_undegraded":"2023-10-21T05:07:20.351624+0000","last_fullsized":"2023-10-21T05:07:20.351624+0000","mapping_epoch":13,"log_start":"80'5000","ondisk_log_start":"80'5000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:36:31.828315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":242,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:05:55.306329+0000","last_change":"2023-10-21T04:01:30.998967+0000","last_active":"2023-10-21T05:05:55.306329+0000","last_peered":"2023-10-21T05:05:55.306329+0000","last_clean":"2023-10-21T05:05:55.306329+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:05:55.306329+0000","last_undegraded":"2023-10-21T05:05:55.306329+0000","last_fullsized":"2023-10-21T05:05:55.306329+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:29.598046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:29.598046+0000","last_clean_scrub_stamp":"2023-10-21T04:01:29.598046+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:21:14.267398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26529493011,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1729119,"num_read_kb":6797740,"num_write":1725157,"num_write_kb":6900622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14133469184,"data_stored":17027440678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40239,"ondisk_log_size":40239,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":35852,"ondisk_log_size":35852,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739162,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8557112,"kb_used_data":6902192,"kb_used_omap":667,"kb_used_meta":1654244,"kb_avail":85814728,"statfs":{"total":96636764160,"available":87874281472,"internally_reserved":0,"allocated":7067844608,"data_stored":8514340631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":683328,"internal_metadata":1693946560},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:31 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.38300000000000001,"15min":0.40000000000000002},"min":{"1min":0.191,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.48899999999999999,"5min":3.7160000000000002,"15min":4.6139999999999999},"last":0.316},{"interface":"front","average":{"1min":0.308,"5min":0.38200000000000001,"15min":0.41899999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.45700000000000002,"5min":3.7749999999999999,"15min":5.0979999999999999},"last":0.26000000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:06:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57799999999999996,"15min":0.70999999999999996},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.17499999999999999},"max":{"1min":3.887,"5min":5.4199999999999999,"15min":31.763999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.502,"15min":0.56999999999999995},"min":{"1min":0.26300000000000001,"5min":0.21199999999999999,"15min":0.20300000000000001},"max":{"1min":3.9129999999999998,"5min":5.2199999999999998,"15min":31.832999999999998},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739159,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4994516,"kb_used_data":3494988,"kb_used_omap":353,"kb_used_meta":1499166,"kb_avail":89377324,"statfs":{"total":96636764160,"available":91522379776,"internally_reserved":0,"allocated":3578867712,"data_stored":4324794217,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1535146199},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,5,4,10,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.44800000000000001,"15min":0.54100000000000004},"min":{"1min":0.20200000000000001,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":3.5219999999999998,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.435,"15min":0.55500000000000005},"min":{"1min":0.22500000000000001,"5min":0.19700000000000001,"15min":0.17499999999999999},"max":{"1min":3.4769999999999999,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:06:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.45400000000000001,"15min":0.41999999999999998},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":4.2729999999999997,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.41799999999999998,"15min":0.42999999999999999},"min":{"1min":0.27800000000000002,"5min":0.23300000000000001,"15min":0.22},"max":{"1min":4.306,"5min":4.306,"15min":5.3380000000000001},"last":0.45600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739162,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4685136,"kb_used_data":3407432,"kb_used_omap":329,"kb_used_meta":1277366,"kb_avail":89686704,"statfs":{"total":96636764160,"available":91839184896,"internally_reserved":0,"allocated":3489210368,"data_stored":4190124858,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1308022794},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34899999999999998,"15min":0.58899999999999997},"min":{"1min":0.20599999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.60099999999999998,"5min":1.2789999999999999,"15min":55.045999999999999},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.63900000000000001},"min":{"1min":0.27600000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.63600000000000001,"5min":1.3,"15min":55.079999999999998},"last":0.30599999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38400000000000001,"15min":0.434},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.19},"max":{"1min":0.66600000000000004,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.254},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.437,"15min":0.47199999999999998},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.69199999999999995,"5min":3.617,"15min":5.4900000000000002},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3488387072,"data_stored":4189323264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3577856000,"data_stored":4323905555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7067226112,"data_stored":8514211859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:07:22.208 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:07:22.763 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:07:22.764 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:07:22.795 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2059,"stamp":"2023-10-21T05:07:22.075786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26530086017,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1748207,"num_read_kb":6814061,"num_write":1725895,"num_write_kb":6902823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76188,"ondisk_log_size":76188,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18236764,"kb_used_data":13804612,"kb_used_omap":1351,"kb_used_meta":4430776,"kb_avail":264878756,"statfs":{"total":289910292480,"available":271235846144,"internally_reserved":0,"allocated":14135922688,"data_stored":17029259706,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1383519,"internal_metadata":4537115553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,4,3,14,7,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18190336,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4765,"num_read_kb":19044,"num_write":5215,"num_write_kb":20860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006957"},"pg_stats":[{"pgid":"2.7","version":"84'12096","reported_seq":16645,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.319897+0000","last_change":"2023-10-21T04:01:35.655104+0000","last_active":"2023-10-21T05:07:20.319897+0000","last_peered":"2023-10-21T05:07:20.319897+0000","last_clean":"2023-10-21T05:07:20.319897+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:07:20.319897+0000","last_undegraded":"2023-10-21T05:07:20.319897+0000","last_fullsized":"2023-10-21T05:07:20.319897+0000","mapping_epoch":13,"log_start":"59'2000","ondisk_log_start":"59'2000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:01:13.743335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"84'6171","reported_seq":9475,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.292008+0000","last_change":"2023-10-21T04:01:35.652553+0000","last_active":"2023-10-21T05:07:20.292008+0000","last_peered":"2023-10-21T05:07:20.292008+0000","last_clean":"2023-10-21T05:07:20.292008+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:07:20.292008+0000","last_undegraded":"2023-10-21T05:07:20.292008+0000","last_fullsized":"2023-10-21T05:07:20.292008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:04:21.394002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"84'3083","reported_seq":4365,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.094379+0000","last_change":"2023-10-21T04:01:35.651529+0000","last_active":"2023-10-21T05:07:20.094379+0000","last_peered":"2023-10-21T05:07:20.094379+0000","last_clean":"2023-10-21T05:07:20.094379+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:07:20.094379+0000","last_undegraded":"2023-10-21T05:07:20.094379+0000","last_fullsized":"2023-10-21T05:07:20.094379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:33:14.743990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"84'6115","reported_seq":8829,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.295321+0000","last_change":"2023-10-21T04:01:35.652367+0000","last_active":"2023-10-21T05:07:20.295321+0000","last_peered":"2023-10-21T05:07:20.295321+0000","last_clean":"2023-10-21T05:07:20.295321+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:07:20.295321+0000","last_undegraded":"2023-10-21T05:07:20.295321+0000","last_fullsized":"2023-10-21T05:07:20.295321+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":6115,"log_dups_size":0,"ondisk_log_size":6115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:07:52.675494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"84'439867","reported_seq":876521,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:18.947939+0000","last_change":"2023-10-21T04:14:04.720485+0000","last_active":"2023-10-21T05:07:18.947939+0000","last_peered":"2023-10-21T05:07:18.947939+0000","last_clean":"2023-10-21T05:07:18.947939+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:07:18.947939+0000","last_undegraded":"2023-10-21T05:07:18.947939+0000","last_fullsized":"2023-10-21T05:07:18.947939+0000","mapping_epoch":16,"log_start":"82'429814","ondisk_log_start":"82'429814","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:20:02.233184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6720344064,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":438140,"num_read_kb":1722828,"num_write":438195,"num_write_kb":1752780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"62'27","reported_seq":191,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:06:33.659035+0000","last_change":"2023-10-21T04:01:35.650136+0000","last_active":"2023-10-21T05:06:33.659035+0000","last_peered":"2023-10-21T05:06:33.659035+0000","last_clean":"2023-10-21T05:06:33.659035+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:06:33.659035+0000","last_undegraded":"2023-10-21T05:06:33.659035+0000","last_fullsized":"2023-10-21T05:06:33.659035+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:55:55.905920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"84'443709","reported_seq":885719,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.729509+0000","last_change":"2023-10-21T04:14:04.706949+0000","last_active":"2023-10-21T05:07:20.729509+0000","last_peered":"2023-10-21T05:07:20.729509+0000","last_clean":"2023-10-21T05:07:20.729509+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:07:20.729509+0000","last_undegraded":"2023-10-21T05:07:20.729509+0000","last_fullsized":"2023-10-21T05:07:20.729509+0000","mapping_epoch":16,"log_start":"82'433618","ondisk_log_start":"82'433618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:21:43.744488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6830796800,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":443575,"num_read_kb":1743896,"num_write":441967,"num_write_kb":1767868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"80'133","reported_seq":3544,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:16.124984+0000","last_change":"2023-10-21T04:01:35.652988+0000","last_active":"2023-10-21T05:07:16.124984+0000","last_peered":"2023-10-21T05:07:16.124984+0000","last_clean":"2023-10-21T05:07:16.124984+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:07:16.124984+0000","last_undegraded":"2023-10-21T05:07:16.124984+0000","last_fullsized":"2023-10-21T05:07:16.124984+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:39:12.668814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"84'415012","reported_seq":829475,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:18.924162+0000","last_change":"2023-10-21T04:14:04.721281+0000","last_active":"2023-10-21T05:07:18.924162+0000","last_peered":"2023-10-21T05:07:18.924162+0000","last_clean":"2023-10-21T05:07:18.924162+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:07:18.924162+0000","last_undegraded":"2023-10-21T05:07:18.924162+0000","last_fullsized":"2023-10-21T05:07:18.924162+0000","mapping_epoch":16,"log_start":"82'404918","ondisk_log_start":"82'404918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:54:59.670291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6332669952,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":415827,"num_read_kb":1634656,"num_write":413461,"num_write_kb":1653844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"80'130","reported_seq":3475,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:15.916316+0000","last_change":"2023-10-21T04:01:35.650390+0000","last_active":"2023-10-21T05:07:15.916316+0000","last_peered":"2023-10-21T05:07:15.916316+0000","last_clean":"2023-10-21T05:07:15.916316+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:07:15.916316+0000","last_undegraded":"2023-10-21T05:07:15.916316+0000","last_fullsized":"2023-10-21T05:07:15.916316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:08:09.428828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"84'433225","reported_seq":863299,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.751989+0000","last_change":"2023-10-21T04:14:04.713203+0000","last_active":"2023-10-21T05:07:20.751989+0000","last_peered":"2023-10-21T05:07:20.751989+0000","last_clean":"2023-10-21T05:07:20.751989+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:07:20.751989+0000","last_undegraded":"2023-10-21T05:07:20.751989+0000","last_fullsized":"2023-10-21T05:07:20.751989+0000","mapping_epoch":16,"log_start":"82'423224","ondisk_log_start":"82'423224","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:49:18.294604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6645682195,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":431577,"num_read_kb":1696360,"num_write":431534,"num_write_kb":1726130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"84'15097","reported_seq":20883,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.351624+0000","last_change":"2023-10-21T04:01:35.650183+0000","last_active":"2023-10-21T05:07:20.351624+0000","last_peered":"2023-10-21T05:07:20.351624+0000","last_clean":"2023-10-21T05:07:20.351624+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:07:20.351624+0000","last_undegraded":"2023-10-21T05:07:20.351624+0000","last_fullsized":"2023-10-21T05:07:20.351624+0000","mapping_epoch":13,"log_start":"80'5000","ondisk_log_start":"80'5000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:36:31.828315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":242,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:05:55.306329+0000","last_change":"2023-10-21T04:01:30.998967+0000","last_active":"2023-10-21T05:05:55.306329+0000","last_peered":"2023-10-21T05:05:55.306329+0000","last_clean":"2023-10-21T05:05:55.306329+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:05:55.306329+0000","last_undegraded":"2023-10-21T05:05:55.306329+0000","last_fullsized":"2023-10-21T05:05:55.306329+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:29.598046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:29.598046+0000","last_clean_scrub_stamp":"2023-10-21T04:01:29.598046+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:21:14.267398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26529493011,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1729119,"num_read_kb":6797740,"num_write":1725157,"num_write_kb":6900622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14133469184,"data_stored":17027440678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40239,"ondisk_log_size":40239,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":35852,"ondisk_log_size":35852,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739162,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8557112,"kb_used_data":6902192,"kb_used_omap":667,"kb_used_meta":1654244,"kb_avail":85814728,"statfs":{"total":96636764160,"available":87874281472,"internally_reserved":0,"allocated":7067844608,"data_stored":8514340631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":683328,"internal_metadata":1693946560},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:31 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.38300000000000001,"15min":0.40000000000000002},"min":{"1min":0.191,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.48899999999999999,"5min":3.7160000000000002,"15min":4.6139999999999999},"last":0.316},{"interface":"front","average":{"1min":0.308,"5min":0.38200000000000001,"15min":0.41899999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.45700000000000002,"5min":3.7749999999999999,"15min":5.0979999999999999},"last":0.26000000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:06:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57799999999999996,"15min":0.70999999999999996},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.17499999999999999},"max":{"1min":3.887,"5min":5.4199999999999999,"15min":31.763999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.502,"15min":0.56999999999999995},"min":{"1min":0.26300000000000001,"5min":0.21199999999999999,"15min":0.20300000000000001},"max":{"1min":3.9129999999999998,"5min":5.2199999999999998,"15min":31.832999999999998},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739159,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4994516,"kb_used_data":3494988,"kb_used_omap":353,"kb_used_meta":1499166,"kb_avail":89377324,"statfs":{"total":96636764160,"available":91522379776,"internally_reserved":0,"allocated":3578867712,"data_stored":4324794217,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1535146199},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,5,4,10,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.44800000000000001,"15min":0.54100000000000004},"min":{"1min":0.20200000000000001,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":3.5219999999999998,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.435,"15min":0.55500000000000005},"min":{"1min":0.22500000000000001,"5min":0.19700000000000001,"15min":0.17499999999999999},"max":{"1min":3.4769999999999999,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:06:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.45400000000000001,"15min":0.41999999999999998},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":4.2729999999999997,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.41799999999999998,"15min":0.42999999999999999},"min":{"1min":0.27800000000000002,"5min":0.23300000000000001,"15min":0.22},"max":{"1min":4.306,"5min":4.306,"15min":5.3380000000000001},"last":0.45600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739162,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4685136,"kb_used_data":3407432,"kb_used_omap":329,"kb_used_meta":1277366,"kb_avail":89686704,"statfs":{"total":96636764160,"available":91839184896,"internally_reserved":0,"allocated":3489210368,"data_stored":4190124858,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1308022794},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34899999999999998,"15min":0.58899999999999997},"min":{"1min":0.20599999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.60099999999999998,"5min":1.2789999999999999,"15min":55.045999999999999},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.63900000000000001},"min":{"1min":0.27600000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.63600000000000001,"5min":1.3,"15min":55.079999999999998},"last":0.30599999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38400000000000001,"15min":0.434},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.19},"max":{"1min":0.66600000000000004,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.254},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.437,"15min":0.47199999999999998},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.69199999999999995,"5min":3.617,"15min":5.4900000000000002},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3488387072,"data_stored":4189323264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3577856000,"data_stored":4323905555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7067226112,"data_stored":8514211859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:07:22.796 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-21T05:07:22.797 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:07:23.273 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:07:23.274 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:07:23.313 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2059,"stamp":"2023-10-21T05:07:22.075786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26530086017,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1748207,"num_read_kb":6814061,"num_write":1725895,"num_write_kb":6902823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76188,"ondisk_log_size":76188,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18236764,"kb_used_data":13804612,"kb_used_omap":1351,"kb_used_meta":4430776,"kb_avail":264878756,"statfs":{"total":289910292480,"available":271235846144,"internally_reserved":0,"allocated":14135922688,"data_stored":17029259706,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1383519,"internal_metadata":4537115553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,4,3,14,7,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18190336,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4765,"num_read_kb":19044,"num_write":5215,"num_write_kb":20860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006957"},"pg_stats":[{"pgid":"2.7","version":"84'12096","reported_seq":16645,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.319897+0000","last_change":"2023-10-21T04:01:35.655104+0000","last_active":"2023-10-21T05:07:20.319897+0000","last_peered":"2023-10-21T05:07:20.319897+0000","last_clean":"2023-10-21T05:07:20.319897+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:07:20.319897+0000","last_undegraded":"2023-10-21T05:07:20.319897+0000","last_fullsized":"2023-10-21T05:07:20.319897+0000","mapping_epoch":13,"log_start":"59'2000","ondisk_log_start":"59'2000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:01:13.743335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"84'6171","reported_seq":9475,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.292008+0000","last_change":"2023-10-21T04:01:35.652553+0000","last_active":"2023-10-21T05:07:20.292008+0000","last_peered":"2023-10-21T05:07:20.292008+0000","last_clean":"2023-10-21T05:07:20.292008+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:07:20.292008+0000","last_undegraded":"2023-10-21T05:07:20.292008+0000","last_fullsized":"2023-10-21T05:07:20.292008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:04:21.394002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"84'3083","reported_seq":4365,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.094379+0000","last_change":"2023-10-21T04:01:35.651529+0000","last_active":"2023-10-21T05:07:20.094379+0000","last_peered":"2023-10-21T05:07:20.094379+0000","last_clean":"2023-10-21T05:07:20.094379+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:07:20.094379+0000","last_undegraded":"2023-10-21T05:07:20.094379+0000","last_fullsized":"2023-10-21T05:07:20.094379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:33:14.743990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"84'6115","reported_seq":8829,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.295321+0000","last_change":"2023-10-21T04:01:35.652367+0000","last_active":"2023-10-21T05:07:20.295321+0000","last_peered":"2023-10-21T05:07:20.295321+0000","last_clean":"2023-10-21T05:07:20.295321+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:07:20.295321+0000","last_undegraded":"2023-10-21T05:07:20.295321+0000","last_fullsized":"2023-10-21T05:07:20.295321+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":6115,"log_dups_size":0,"ondisk_log_size":6115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:07:52.675494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"84'439867","reported_seq":876521,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:18.947939+0000","last_change":"2023-10-21T04:14:04.720485+0000","last_active":"2023-10-21T05:07:18.947939+0000","last_peered":"2023-10-21T05:07:18.947939+0000","last_clean":"2023-10-21T05:07:18.947939+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:07:18.947939+0000","last_undegraded":"2023-10-21T05:07:18.947939+0000","last_fullsized":"2023-10-21T05:07:18.947939+0000","mapping_epoch":16,"log_start":"82'429814","ondisk_log_start":"82'429814","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:20:02.233184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6720344064,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":438140,"num_read_kb":1722828,"num_write":438195,"num_write_kb":1752780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"62'27","reported_seq":191,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:06:33.659035+0000","last_change":"2023-10-21T04:01:35.650136+0000","last_active":"2023-10-21T05:06:33.659035+0000","last_peered":"2023-10-21T05:06:33.659035+0000","last_clean":"2023-10-21T05:06:33.659035+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:06:33.659035+0000","last_undegraded":"2023-10-21T05:06:33.659035+0000","last_fullsized":"2023-10-21T05:06:33.659035+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:55:55.905920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"84'443709","reported_seq":885719,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.729509+0000","last_change":"2023-10-21T04:14:04.706949+0000","last_active":"2023-10-21T05:07:20.729509+0000","last_peered":"2023-10-21T05:07:20.729509+0000","last_clean":"2023-10-21T05:07:20.729509+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:07:20.729509+0000","last_undegraded":"2023-10-21T05:07:20.729509+0000","last_fullsized":"2023-10-21T05:07:20.729509+0000","mapping_epoch":16,"log_start":"82'433618","ondisk_log_start":"82'433618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:21:43.744488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6830796800,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":443575,"num_read_kb":1743896,"num_write":441967,"num_write_kb":1767868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"80'133","reported_seq":3544,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:16.124984+0000","last_change":"2023-10-21T04:01:35.652988+0000","last_active":"2023-10-21T05:07:16.124984+0000","last_peered":"2023-10-21T05:07:16.124984+0000","last_clean":"2023-10-21T05:07:16.124984+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:07:16.124984+0000","last_undegraded":"2023-10-21T05:07:16.124984+0000","last_fullsized":"2023-10-21T05:07:16.124984+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:39:12.668814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"84'415012","reported_seq":829475,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:18.924162+0000","last_change":"2023-10-21T04:14:04.721281+0000","last_active":"2023-10-21T05:07:18.924162+0000","last_peered":"2023-10-21T05:07:18.924162+0000","last_clean":"2023-10-21T05:07:18.924162+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:07:18.924162+0000","last_undegraded":"2023-10-21T05:07:18.924162+0000","last_fullsized":"2023-10-21T05:07:18.924162+0000","mapping_epoch":16,"log_start":"82'404918","ondisk_log_start":"82'404918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:54:59.670291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6332669952,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":415827,"num_read_kb":1634656,"num_write":413461,"num_write_kb":1653844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"80'130","reported_seq":3475,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:15.916316+0000","last_change":"2023-10-21T04:01:35.650390+0000","last_active":"2023-10-21T05:07:15.916316+0000","last_peered":"2023-10-21T05:07:15.916316+0000","last_clean":"2023-10-21T05:07:15.916316+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:07:15.916316+0000","last_undegraded":"2023-10-21T05:07:15.916316+0000","last_fullsized":"2023-10-21T05:07:15.916316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:08:09.428828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"84'433225","reported_seq":863299,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.751989+0000","last_change":"2023-10-21T04:14:04.713203+0000","last_active":"2023-10-21T05:07:20.751989+0000","last_peered":"2023-10-21T05:07:20.751989+0000","last_clean":"2023-10-21T05:07:20.751989+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:07:20.751989+0000","last_undegraded":"2023-10-21T05:07:20.751989+0000","last_fullsized":"2023-10-21T05:07:20.751989+0000","mapping_epoch":16,"log_start":"82'423224","ondisk_log_start":"82'423224","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:49:18.294604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6645682195,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":431577,"num_read_kb":1696360,"num_write":431534,"num_write_kb":1726130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"84'15097","reported_seq":20883,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:20.351624+0000","last_change":"2023-10-21T04:01:35.650183+0000","last_active":"2023-10-21T05:07:20.351624+0000","last_peered":"2023-10-21T05:07:20.351624+0000","last_clean":"2023-10-21T05:07:20.351624+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:07:20.351624+0000","last_undegraded":"2023-10-21T05:07:20.351624+0000","last_fullsized":"2023-10-21T05:07:20.351624+0000","mapping_epoch":13,"log_start":"80'5000","ondisk_log_start":"80'5000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:36:31.828315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":242,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:05:55.306329+0000","last_change":"2023-10-21T04:01:30.998967+0000","last_active":"2023-10-21T05:05:55.306329+0000","last_peered":"2023-10-21T05:05:55.306329+0000","last_clean":"2023-10-21T05:05:55.306329+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:05:55.306329+0000","last_undegraded":"2023-10-21T05:05:55.306329+0000","last_fullsized":"2023-10-21T05:05:55.306329+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:29.598046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:29.598046+0000","last_clean_scrub_stamp":"2023-10-21T04:01:29.598046+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:21:14.267398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26529493011,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1729119,"num_read_kb":6797740,"num_write":1725157,"num_write_kb":6900622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14133469184,"data_stored":17027440678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40239,"ondisk_log_size":40239,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":35852,"ondisk_log_size":35852,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739162,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8557112,"kb_used_data":6902192,"kb_used_omap":667,"kb_used_meta":1654244,"kb_avail":85814728,"statfs":{"total":96636764160,"available":87874281472,"internally_reserved":0,"allocated":7067844608,"data_stored":8514340631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":683328,"internal_metadata":1693946560},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:31 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.38300000000000001,"15min":0.40000000000000002},"min":{"1min":0.191,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.48899999999999999,"5min":3.7160000000000002,"15min":4.6139999999999999},"last":0.316},{"interface":"front","average":{"1min":0.308,"5min":0.38200000000000001,"15min":0.41899999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.45700000000000002,"5min":3.7749999999999999,"15min":5.0979999999999999},"last":0.26000000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:06:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57799999999999996,"15min":0.70999999999999996},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.17499999999999999},"max":{"1min":3.887,"5min":5.4199999999999999,"15min":31.763999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.502,"15min":0.56999999999999995},"min":{"1min":0.26300000000000001,"5min":0.21199999999999999,"15min":0.20300000000000001},"max":{"1min":3.9129999999999998,"5min":5.2199999999999998,"15min":31.832999999999998},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739159,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4994516,"kb_used_data":3494988,"kb_used_omap":353,"kb_used_meta":1499166,"kb_avail":89377324,"statfs":{"total":96636764160,"available":91522379776,"internally_reserved":0,"allocated":3578867712,"data_stored":4324794217,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1535146199},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,5,4,10,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.44800000000000001,"15min":0.54100000000000004},"min":{"1min":0.20200000000000001,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":3.5219999999999998,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.435,"15min":0.55500000000000005},"min":{"1min":0.22500000000000001,"5min":0.19700000000000001,"15min":0.17499999999999999},"max":{"1min":3.4769999999999999,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:06:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.45400000000000001,"15min":0.41999999999999998},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":4.2729999999999997,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.41799999999999998,"15min":0.42999999999999999},"min":{"1min":0.27800000000000002,"5min":0.23300000000000001,"15min":0.22},"max":{"1min":4.306,"5min":4.306,"15min":5.3380000000000001},"last":0.45600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739162,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4685136,"kb_used_data":3407432,"kb_used_omap":329,"kb_used_meta":1277366,"kb_avail":89686704,"statfs":{"total":96636764160,"available":91839184896,"internally_reserved":0,"allocated":3489210368,"data_stored":4190124858,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1308022794},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34899999999999998,"15min":0.58899999999999997},"min":{"1min":0.20599999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.60099999999999998,"5min":1.2789999999999999,"15min":55.045999999999999},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.63900000000000001},"min":{"1min":0.27600000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.63600000000000001,"5min":1.3,"15min":55.079999999999998},"last":0.30599999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38400000000000001,"15min":0.434},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.19},"max":{"1min":0.66600000000000004,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.254},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.437,"15min":0.47199999999999998},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.69199999999999995,"5min":3.617,"15min":5.4900000000000002},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3488387072,"data_stored":4189323264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3577856000,"data_stored":4323905555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7067226112,"data_stored":8514211859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:07:23.315 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-21T05:07:23.831 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:07:23.831 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":84,"fsid":"f3f3d351-07e7-41e0-8ea5-8c6ad08b8ea3","created":"2023-10-21T04:01:18.489015+0000","modified":"2023-10-21T05:07:04.696097+0000","last_up_change":"2023-10-21T04:01:27.594123+0000","last_in_change":"2023-10-21T04:01:19.406189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-21T04:01:29.307259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-21T04:01:33.690298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-10-21T04:01:39.716330+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":71,"snap_epoch":66,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"096797d6-e2df-4521-8001-08e482bbe25c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6817","nonce":2130934728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6819","nonce":2130934728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6823","nonce":2130934728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":2130934728},{"type":"v1","addr":"172.21.15.133:6821","nonce":2130934728}]},"public_addr":"172.21.15.133:6817/2130934728","cluster_addr":"172.21.15.133:6819/2130934728","heartbeat_back_addr":"172.21.15.133:6823/2130934728","heartbeat_front_addr":"172.21.15.133:6821/2130934728","state":["exists","up"]},{"osd":1,"uuid":"7229425e-3c45-47b8-8e9e-d2c98d06a01f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6809","nonce":3846075335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6811","nonce":3846075335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6815","nonce":3846075335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3846075335},{"type":"v1","addr":"172.21.15.133:6813","nonce":3846075335}]},"public_addr":"172.21.15.133:6809/3846075335","cluster_addr":"172.21.15.133:6811/3846075335","heartbeat_back_addr":"172.21.15.133:6815/3846075335","heartbeat_front_addr":"172.21.15.133:6813/3846075335","state":["exists","up"]},{"osd":2,"uuid":"dfe1a0ba-0132-49bb-b531-389d265829eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6801","nonce":350344564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6803","nonce":350344564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6807","nonce":350344564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":350344564},{"type":"v1","addr":"172.21.15.133:6805","nonce":350344564}]},"public_addr":"172.21.15.133:6801/350344564","cluster_addr":"172.21.15.133:6803/350344564","heartbeat_back_addr":"172.21.15.133:6807/350344564","heartbeat_front_addr":"172.21.15.133:6805/350344564","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:01:24.866966+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:01:24.660125+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-21T04:01:24.535867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.133:0/3061939021":"2023-10-21T05:20:56.486689+0000","172.21.15.133:0/2789602076":"2023-10-21T05:19:47.360038+0000","172.21.15.133:0/374453278":"2023-10-21T05:08:09.557706+0000","172.21.15.133:0/210740177":"2023-10-21T05:09:17.505269+0000","172.21.15.133:0/3729849345":"2023-10-21T05:18:37.972558+0000","172.21.15.133:0/1019917016":"2023-10-21T05:16:20.748476+0000","172.21.15.133:0/1058636164":"2023-10-21T05:10:25.520900+0000","172.21.15.133:0/4278077090":"2023-10-21T05:11:44.615111+0000","172.21.15.133:0/102508650":"2023-10-21T05:12:43.026525+0000","172.21.15.133:0/2426556961":"2023-10-21T05:15:11.780509+0000","172.21.15.133:0/3237579704":"2023-10-21T05:17:29.619775+0000","172.21.15.133:0/2752721599":"2023-10-21T05:14:02.696045+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-21T05:07:24.864 INFO:tasks.ceph:Scrubbing osd.0 2023-10-21T05:07:24.865 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-21T05:07:25.224 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2023-10-21T05:07:25.224 INFO:teuthology.orchestra.run.smithi133.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-21T05:07:25.225 INFO:teuthology.orchestra.run.smithi133.stdout:} 2023-10-21T05:07:25.267 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-21T05:07:25.925 INFO:teuthology.orchestra.run.smithi133.stderr:instructed osd(s) 0 to deep-scrub 2023-10-21T05:07:25.973 INFO:tasks.ceph:Scrubbing osd.1 2023-10-21T05:07:25.974 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-21T05:07:26.348 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2023-10-21T05:07:26.349 INFO:teuthology.orchestra.run.smithi133.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-21T05:07:26.349 INFO:teuthology.orchestra.run.smithi133.stdout:} 2023-10-21T05:07:26.406 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-21T05:07:26.903 INFO:teuthology.orchestra.run.smithi133.stderr:instructed osd(s) 1 to deep-scrub 2023-10-21T05:07:26.944 INFO:tasks.ceph:Scrubbing osd.2 2023-10-21T05:07:26.944 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-21T05:07:27.291 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2023-10-21T05:07:27.291 INFO:teuthology.orchestra.run.smithi133.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-21T05:07:27.291 INFO:teuthology.orchestra.run.smithi133.stdout:} 2023-10-21T05:07:27.316 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-21T05:07:27.878 INFO:teuthology.orchestra.run.smithi133.stderr:instructed osd(s) 2 to deep-scrub 2023-10-21T05:07:27.907 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:07:28.511 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:07:28.512 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:07:28.561 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2062,"stamp":"2023-10-21T05:07:28.076759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26536799361,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1750019,"num_read_kb":6821281,"num_write":1727802,"num_write_kb":6910451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76051,"ondisk_log_size":76051,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18251152,"kb_used_data":13818996,"kb_used_omap":1351,"kb_used_meta":4430776,"kb_avail":264864368,"statfs":{"total":289910292480,"available":271221112832,"internally_reserved":0,"allocated":14150651904,"data_stored":17043988922,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1383519,"internal_metadata":4537115553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,7,16,12,14,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14016512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3721,"num_read_kb":14856,"num_write":4003,"num_write_kb":16012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002020"},"pg_stats":[{"pgid":"2.7","version":"84'12112","reported_seq":16665,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:24.863488+0000","last_change":"2023-10-21T04:01:35.655104+0000","last_active":"2023-10-21T05:07:24.863488+0000","last_peered":"2023-10-21T05:07:24.863488+0000","last_clean":"2023-10-21T05:07:24.863488+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:07:24.863488+0000","last_undegraded":"2023-10-21T05:07:24.863488+0000","last_fullsized":"2023-10-21T05:07:24.863488+0000","mapping_epoch":13,"log_start":"59'2100","ondisk_log_start":"59'2100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:01:13.743335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"84'6179","reported_seq":9485,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:24.848939+0000","last_change":"2023-10-21T04:01:35.652553+0000","last_active":"2023-10-21T05:07:24.848939+0000","last_peered":"2023-10-21T05:07:24.848939+0000","last_clean":"2023-10-21T05:07:24.848939+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:07:24.848939+0000","last_undegraded":"2023-10-21T05:07:24.848939+0000","last_fullsized":"2023-10-21T05:07:24.848939+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T04:04:21.394002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"84'3087","reported_seq":4370,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:24.351964+0000","last_change":"2023-10-21T04:01:35.651529+0000","last_active":"2023-10-21T05:07:24.351964+0000","last_peered":"2023-10-21T05:07:24.351964+0000","last_clean":"2023-10-21T05:07:24.351964+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:07:24.351964+0000","last_undegraded":"2023-10-21T05:07:24.351964+0000","last_fullsized":"2023-10-21T05:07:24.351964+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":3087,"log_dups_size":0,"ondisk_log_size":3087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T07:33:14.743990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"84'6123","reported_seq":8839,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:24.845468+0000","last_change":"2023-10-21T04:01:35.652367+0000","last_active":"2023-10-21T05:07:24.845468+0000","last_peered":"2023-10-21T05:07:24.845468+0000","last_clean":"2023-10-21T05:07:24.845468+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:07:24.845468+0000","last_undegraded":"2023-10-21T05:07:24.845468+0000","last_fullsized":"2023-10-21T05:07:24.845468+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:07:52.675494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"84'440339","reported_seq":877457,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:23.958009+0000","last_change":"2023-10-21T04:14:04.720485+0000","last_active":"2023-10-21T05:07:23.958009+0000","last_peered":"2023-10-21T05:07:23.958009+0000","last_clean":"2023-10-21T05:07:23.958009+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:07:23.958009+0000","last_undegraded":"2023-10-21T05:07:23.958009+0000","last_fullsized":"2023-10-21T05:07:23.958009+0000","mapping_epoch":16,"log_start":"83'430314","ondisk_log_start":"83'430314","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T05:20:02.233184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6721974272,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":438602,"num_read_kb":1724652,"num_write":438667,"num_write_kb":1754668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"62'27","reported_seq":191,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:06:33.659035+0000","last_change":"2023-10-21T04:01:35.650136+0000","last_active":"2023-10-21T05:06:33.659035+0000","last_peered":"2023-10-21T05:06:33.659035+0000","last_clean":"2023-10-21T05:06:33.659035+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:06:33.659035+0000","last_undegraded":"2023-10-21T05:06:33.659035+0000","last_fullsized":"2023-10-21T05:06:33.659035+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:55:55.905920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"84'444193","reported_seq":886692,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:25.776512+0000","last_change":"2023-10-21T04:14:04.706949+0000","last_active":"2023-10-21T05:07:25.776512+0000","last_peered":"2023-10-21T05:07:25.776512+0000","last_clean":"2023-10-21T05:07:25.776512+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:07:25.776512+0000","last_undegraded":"2023-10-21T05:07:25.776512+0000","last_fullsized":"2023-10-21T05:07:25.776512+0000","mapping_epoch":16,"log_start":"83'434118","ondisk_log_start":"83'434118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:21:43.744488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6832472064,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":444053,"num_read_kb":1745808,"num_write":442451,"num_write_kb":1769804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"80'133","reported_seq":3547,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:21.034210+0000","last_change":"2023-10-21T04:01:35.652988+0000","last_active":"2023-10-21T05:07:21.034210+0000","last_peered":"2023-10-21T05:07:21.034210+0000","last_clean":"2023-10-21T05:07:21.034210+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:07:21.034210+0000","last_undegraded":"2023-10-21T05:07:21.034210+0000","last_fullsized":"2023-10-21T05:07:21.034210+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:39:12.668814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"84'415469","reported_seq":830347,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:23.963171+0000","last_change":"2023-10-21T04:14:04.721281+0000","last_active":"2023-10-21T05:07:23.963171+0000","last_peered":"2023-10-21T05:07:23.963171+0000","last_clean":"2023-10-21T05:07:23.963171+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:07:23.963171+0000","last_undegraded":"2023-10-21T05:07:23.963171+0000","last_fullsized":"2023-10-21T05:07:23.963171+0000","mapping_epoch":16,"log_start":"82'405418","ondisk_log_start":"82'405418","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:54:59.670291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6334320640,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":416244,"num_read_kb":1636324,"num_write":413918,"num_write_kb":1655672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"80'130","reported_seq":3478,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:21.039980+0000","last_change":"2023-10-21T04:01:35.650390+0000","last_active":"2023-10-21T05:07:21.039980+0000","last_peered":"2023-10-21T05:07:21.039980+0000","last_clean":"2023-10-21T05:07:21.039980+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:07:21.039980+0000","last_undegraded":"2023-10-21T05:07:21.039980+0000","last_fullsized":"2023-10-21T05:07:21.039980+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:08:09.428828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"84'433719","reported_seq":864247,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:25.770825+0000","last_change":"2023-10-21T04:14:04.713203+0000","last_active":"2023-10-21T05:07:25.770825+0000","last_peered":"2023-10-21T05:07:25.770825+0000","last_clean":"2023-10-21T05:07:25.770825+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:07:25.770825+0000","last_undegraded":"2023-10-21T05:07:25.770825+0000","last_fullsized":"2023-10-21T05:07:25.770825+0000","mapping_epoch":16,"log_start":"83'423624","ondisk_log_start":"83'423624","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T09:49:18.294604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6647439379,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":432032,"num_read_kb":1698176,"num_write":432028,"num_write_kb":1728106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"84'15117","reported_seq":20908,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:24.888674+0000","last_change":"2023-10-21T04:01:35.650183+0000","last_active":"2023-10-21T05:07:24.888674+0000","last_peered":"2023-10-21T05:07:24.888674+0000","last_clean":"2023-10-21T05:07:24.888674+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:07:24.888674+0000","last_undegraded":"2023-10-21T05:07:24.888674+0000","last_fullsized":"2023-10-21T05:07:24.888674+0000","mapping_epoch":13,"log_start":"80'5100","ondisk_log_start":"80'5100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T14:36:31.828315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":242,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-21T05:05:55.306329+0000","last_change":"2023-10-21T04:01:30.998967+0000","last_active":"2023-10-21T05:05:55.306329+0000","last_peered":"2023-10-21T05:05:55.306329+0000","last_clean":"2023-10-21T05:05:55.306329+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:05:55.306329+0000","last_undegraded":"2023-10-21T05:05:55.306329+0000","last_fullsized":"2023-10-21T05:05:55.306329+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:29.598046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:29.598046+0000","last_clean_scrub_stamp":"2023-10-21T04:01:29.598046+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T11:21:14.267398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26536206355,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1730931,"num_read_kb":6804960,"num_write":1727064,"num_write_kb":6908250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14149210112,"data_stored":17043181606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40246,"ondisk_log_size":40246,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":35708,"ondisk_log_size":35708,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739163,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8556344,"kb_used_data":6909740,"kb_used_omap":667,"kb_used_meta":1645924,"kb_avail":85815496,"statfs":{"total":96636764160,"available":87875067904,"internally_reserved":0,"allocated":7075573760,"data_stored":8522069783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":683328,"internal_metadata":1685426880},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:31 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.38300000000000001,"15min":0.40000000000000002},"min":{"1min":0.191,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.48899999999999999,"5min":3.7160000000000002,"15min":4.6139999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.308,"5min":0.38200000000000001,"15min":0.41899999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.45700000000000002,"5min":3.7749999999999999,"15min":5.0979999999999999},"last":0.28299999999999997}]},{"osd":1,"last update":"Sat Oct 21 05:06:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57799999999999996,"15min":0.70999999999999996},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.17499999999999999},"max":{"1min":3.887,"5min":5.4199999999999999,"15min":31.763999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.502,"15min":0.56999999999999995},"min":{"1min":0.26300000000000001,"5min":0.21199999999999999,"15min":0.20300000000000001},"max":{"1min":3.9129999999999998,"5min":5.2199999999999998,"15min":31.832999999999998},"last":0.434}]}]},{"osd":1,"up_from":8,"seq":34359739160,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5001840,"kb_used_data":3498152,"kb_used_omap":353,"kb_used_meta":1503326,"kb_avail":89370000,"statfs":{"total":96636764160,"available":91514880000,"internally_reserved":0,"allocated":3582107648,"data_stored":4328034153,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1539406039},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,12,4,5,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:06:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.44800000000000001,"15min":0.54100000000000004},"min":{"1min":0.20200000000000001,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":3.5219999999999998,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.435,"15min":0.55500000000000005},"min":{"1min":0.22500000000000001,"5min":0.19700000000000001,"15min":0.17499999999999999},"max":{"1min":3.4769999999999999,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":0.315}]},{"osd":2,"last update":"Sat Oct 21 05:06:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.45400000000000001,"15min":0.41999999999999998},"min":{"1min":0.20699999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":4.2729999999999997,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.41799999999999998,"15min":0.42999999999999999},"min":{"1min":0.27800000000000002,"5min":0.23300000000000001,"15min":0.22},"max":{"1min":4.306,"5min":4.306,"15min":5.3380000000000001},"last":0.35599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739163,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4692968,"kb_used_data":3411104,"kb_used_omap":329,"kb_used_meta":1281526,"kb_avail":89678872,"statfs":{"total":96636764160,"available":91831164928,"internally_reserved":0,"allocated":3492970496,"data_stored":4193884986,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1312282634},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,5,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34899999999999998,"15min":0.58899999999999997},"min":{"1min":0.20599999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.60099999999999998,"5min":1.2789999999999999,"15min":55.045999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.63900000000000001},"min":{"1min":0.27600000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.63600000000000001,"5min":1.3,"15min":55.079999999999998},"last":0.39800000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:06:39 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38400000000000001,"15min":0.434},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.19},"max":{"1min":0.66600000000000004,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.371},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.437,"15min":0.47199999999999998},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.69199999999999995,"5min":3.617,"15min":5.4900000000000002},"last":0.42699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3492200448,"data_stored":4193136640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3581861888,"data_stored":4327911443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7075147776,"data_stored":8522133523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:07:28.564 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-21T04:01:34.622627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.565 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-21T04:01:34.622627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.565 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-21T04:01:34.622627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.565 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-21T04:01:34.622627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.566 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.566 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-21T04:01:34.622627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.566 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.567 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-21T04:01:34.622627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.567 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.567 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-21T04:01:34.622627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.568 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.568 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-21T04:01:34.622627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.568 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-21T04:01:29.598046+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=29, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:28.569 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:07:48.570 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:07:49.080 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:07:49.081 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:07:49.114 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2072,"stamp":"2023-10-21T05:07:48.080762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26563161217,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1756956,"num_read_kb":6848969,"num_write":1735358,"num_write_kb":6940675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12670,"num_omap_keys":284,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76312,"ondisk_log_size":76312,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18419604,"kb_used_data":13879608,"kb_used_omap":1351,"kb_used_meta":4538616,"kb_avail":264695916,"statfs":{"total":289910292480,"available":271048617984,"internally_reserved":0,"allocated":14212718592,"data_stored":17106055610,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1383519,"internal_metadata":4647543713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,4,16,21,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12222464,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3195,"num_read_kb":12756,"num_write":3505,"num_write_kb":14020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4661,"num_omap_keys":130,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002628"},"pg_stats":[{"pgid":"2.7","version":"84'12144","reported_seq":16714,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:45.632400+0000","last_change":"2023-10-21T04:01:35.655104+0000","last_active":"2023-10-21T05:07:45.632400+0000","last_peered":"2023-10-21T05:07:45.632400+0000","last_clean":"2023-10-21T05:07:45.632400+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:07:45.632400+0000","last_undegraded":"2023-10-21T05:07:45.632400+0000","last_fullsized":"2023-10-21T05:07:45.632400+0000","mapping_epoch":13,"log_start":"59'2100","ondisk_log_start":"59'2100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:34.622627+0000","last_clean_scrub_stamp":"2023-10-21T04:01:34.622627+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"84'6204","reported_seq":9527,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:45.613038+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:07:45.613038+0000","last_peered":"2023-10-21T05:07:45.613038+0000","last_clean":"2023-10-21T05:07:45.613038+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:07:45.613038+0000","last_undegraded":"2023-10-21T05:07:45.613038+0000","last_fullsized":"2023-10-21T05:07:45.613038+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6204,"log_dups_size":0,"ondisk_log_size":6204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"84'3107","reported_seq":4403,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:43.892185+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:07:43.892185+0000","last_peered":"2023-10-21T05:07:43.892185+0000","last_clean":"2023-10-21T05:07:43.892185+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:07:43.892185+0000","last_undegraded":"2023-10-21T05:07:43.892185+0000","last_fullsized":"2023-10-21T05:07:43.892185+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"84'6154","reported_seq":8887,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:45.612947+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:07:45.612947+0000","last_peered":"2023-10-21T05:07:45.612947+0000","last_clean":"2023-10-21T05:07:45.612947+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:07:45.612947+0000","last_undegraded":"2023-10-21T05:07:45.612947+0000","last_fullsized":"2023-10-21T05:07:45.612947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6154,"log_dups_size":0,"ondisk_log_size":6154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"84'442266","reported_seq":881136,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:07:44.112441+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:07:44.112441+0000","last_peered":"2023-10-21T05:07:44.112441+0000","last_clean":"2023-10-21T05:07:44.112441+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:07:44.112441+0000","last_undegraded":"2023-10-21T05:07:44.112441+0000","last_fullsized":"2023-10-21T05:07:44.112441+0000","mapping_epoch":16,"log_start":"83'432214","ondisk_log_start":"83'432214","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":5,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 4s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6728626176,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":440350,"num_read_kb":1731632,"num_write":440591,"num_write_kb":1762364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":208,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:30.895797+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:07:30.895797+0000","last_peered":"2023-10-21T05:07:30.895797+0000","last_clean":"2023-10-21T05:07:30.895797+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:07:30.895797+0000","last_undegraded":"2023-10-21T05:07:30.895797+0000","last_fullsized":"2023-10-21T05:07:30.895797+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"84'446111","reported_seq":890383,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:45.912486+0000","last_change":"2023-10-21T04:14:04.706949+0000","last_active":"2023-10-21T05:07:45.912486+0000","last_peered":"2023-10-21T05:07:45.912486+0000","last_clean":"2023-10-21T05:07:45.912486+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:07:45.912486+0000","last_undegraded":"2023-10-21T05:07:45.912486+0000","last_fullsized":"2023-10-21T05:07:45.912486+0000","mapping_epoch":16,"log_start":"83'436018","ondisk_log_start":"83'436018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6839377920,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":445825,"num_read_kb":1752872,"num_write":444369,"num_write_kb":1777476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3576,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:41.518643+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:07:41.518643+0000","last_peered":"2023-10-21T05:07:41.518643+0000","last_clean":"2023-10-21T05:07:41.518643+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:07:41.518643+0000","last_undegraded":"2023-10-21T05:07:41.518643+0000","last_fullsized":"2023-10-21T05:07:41.518643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"84'417295","reported_seq":833824,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:07:44.092698+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:07:44.092698+0000","last_peered":"2023-10-21T05:07:44.092698+0000","last_clean":"2023-10-21T05:07:44.092698+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:07:44.092698+0000","last_undegraded":"2023-10-21T05:07:44.092698+0000","last_fullsized":"2023-10-21T05:07:44.092698+0000","mapping_epoch":16,"log_start":"83'407218","ondisk_log_start":"83'407218","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":16,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 8s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6340517888,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":417886,"num_read_kb":1642872,"num_write":415736,"num_write_kb":1662944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3511,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:41.518409+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:07:41.518409+0000","last_peered":"2023-10-21T05:07:41.518409+0000","last_clean":"2023-10-21T05:07:41.518409+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:07:41.518409+0000","last_undegraded":"2023-10-21T05:07:41.518409+0000","last_fullsized":"2023-10-21T05:07:41.518409+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"84'435615","reported_seq":867918,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:45.913768+0000","last_change":"2023-10-21T04:14:04.713203+0000","last_active":"2023-10-21T05:07:45.913768+0000","last_peered":"2023-10-21T05:07:45.913768+0000","last_clean":"2023-10-21T05:07:45.913768+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:07:45.913768+0000","last_undegraded":"2023-10-21T05:07:45.913768+0000","last_fullsized":"2023-10-21T05:07:45.913768+0000","mapping_epoch":16,"log_start":"83'425524","ondisk_log_start":"83'425524","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6654046227,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":433807,"num_read_kb":1705272,"num_write":433924,"num_write_kb":1735690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"84'15173","reported_seq":20993,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:45.668241+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:07:45.668241+0000","last_peered":"2023-10-21T05:07:45.668241+0000","last_clean":"2023-10-21T05:07:45.668241+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:07:45.668241+0000","last_undegraded":"2023-10-21T05:07:45.668241+0000","last_fullsized":"2023-10-21T05:07:45.668241+0000","mapping_epoch":13,"log_start":"80'5100","ondisk_log_start":"80'5100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":253,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:28.869586+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:07:28.869586+0000","last_peered":"2023-10-21T05:07:28.869586+0000","last_clean":"2023-10-21T05:07:28.869586+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:07:28.869586+0000","last_undegraded":"2023-10-21T05:07:28.869586+0000","last_fullsized":"2023-10-21T05:07:28.869586+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26562568211,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1737868,"num_read_kb":6832648,"num_write":1734620,"num_write_kb":6938474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14210924544,"data_stored":17104896038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40313,"ondisk_log_size":40313,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12670,"num_omap_keys":284,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":35900,"ondisk_log_size":35900,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739167,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8615288,"kb_used_data":6939696,"kb_used_omap":667,"kb_used_meta":1674916,"kb_avail":85756552,"statfs":{"total":96636764160,"available":87814709248,"internally_reserved":0,"allocated":7106248704,"data_stored":8552744727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":683328,"internal_metadata":1715114688},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:07:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.40400000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.51300000000000001,"5min":3.7160000000000002,"15min":4.6139999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.373,"15min":0.41999999999999998},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":0.54900000000000004,"5min":3.7749999999999999,"15min":5.0979999999999999},"last":0.30099999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:07:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60199999999999998,"15min":0.72099999999999997},"min":{"1min":0.20499999999999999,"5min":0.192,"15min":0.17499999999999999},"max":{"1min":4.6950000000000003,"5min":5.4199999999999999,"15min":31.763999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.50700000000000001,"15min":0.56999999999999995},"min":{"1min":0.26600000000000001,"5min":0.219,"15min":0.20300000000000001},"max":{"1min":0.53100000000000003,"5min":5.2199999999999998,"15min":31.832999999999998},"last":0.36499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739164,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5078420,"kb_used_data":3513932,"kb_used_omap":353,"kb_used_meta":1564126,"kb_avail":89293420,"statfs":{"total":96636764160,"available":91436462080,"internally_reserved":0,"allocated":3598266368,"data_stored":4344192873,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1601665239},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,9,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:07:34 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47299999999999998,"15min":0.53700000000000003},"min":{"1min":0.17699999999999999,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":2.6030000000000002,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.42999999999999999,"15min":0.54100000000000004},"min":{"1min":0.24099999999999999,"5min":0.19800000000000001,"15min":0.17499999999999999},"max":{"1min":0.60099999999999998,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":0.41899999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:07:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.49299999999999999,"15min":0.434},"min":{"1min":0.217,"5min":0.20399999999999999,"15min":0.19800000000000001},"max":{"1min":2.6440000000000001,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.437,"15min":0.433},"min":{"1min":0.28599999999999998,"5min":0.23300000000000001,"15min":0.22},"max":{"1min":1.0620000000000001,"5min":4.306,"15min":5.3380000000000001},"last":0.38700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739167,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4725896,"kb_used_data":3425980,"kb_used_omap":329,"kb_used_meta":1299574,"kb_avail":89645944,"statfs":{"total":96636764160,"available":91797446656,"internally_reserved":0,"allocated":3508203520,"data_stored":4209118010,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1330763786},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,5,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.35099999999999998,"15min":0.57299999999999995},"min":{"1min":0.19500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":55.045999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40999999999999998,"15min":0.626},"min":{"1min":0.30599999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":55.079999999999998},"last":0.67500000000000004}]},{"osd":2,"last update":"Sat Oct 21 05:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.39500000000000002,"15min":0.438},"min":{"1min":0.254,"5min":0.221,"15min":0.19},"max":{"1min":0.76400000000000001,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41599999999999998,"15min":0.47399999999999998},"min":{"1min":0.28100000000000003,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.67400000000000004,"5min":1.71,"15min":5.4900000000000002},"last":0.64200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3507515392,"data_stored":4208451584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3597500416,"data_stored":4343549971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7105908736,"data_stored":8552894483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:07:49.116 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-21T04:01:34.622627+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:49.116 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:49.117 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:49.117 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:49.117 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:07:49.118 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:08:09.119 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:08:09.618 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:08:09.619 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:08:09.656 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2082,"stamp":"2023-10-21T05:08:08.084184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26588265601,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1763929,"num_read_kb":6876777,"num_write":1742496,"num_write_kb":6969227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76098,"ondisk_log_size":76098,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18609560,"kb_used_data":13936508,"kb_used_omap":1351,"kb_used_meta":4671672,"kb_avail":264505960,"statfs":{"total":289910292480,"available":270854103040,"internally_reserved":0,"allocated":14270984192,"data_stored":17164321210,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1383519,"internal_metadata":4783793057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,5,7,13,9,13,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12226560,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3374,"num_read_kb":13452,"num_write":3495,"num_write_kb":13980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002088"},"pg_stats":[{"pgid":"2.7","version":"84'12205","reported_seq":16799,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:08:05.737985+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:08:05.737985+0000","last_peered":"2023-10-21T05:08:05.737985+0000","last_clean":"2023-10-21T05:08:05.737985+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:08:05.737985+0000","last_undegraded":"2023-10-21T05:08:05.737985+0000","last_fullsized":"2023-10-21T05:08:05.737985+0000","mapping_epoch":13,"log_start":"61'2200","ondisk_log_start":"61'2200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"84'6230","reported_seq":9561,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:08:05.719657+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:08:05.719657+0000","last_peered":"2023-10-21T05:08:05.719657+0000","last_clean":"2023-10-21T05:08:05.719657+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:08:05.719657+0000","last_undegraded":"2023-10-21T05:08:05.719657+0000","last_fullsized":"2023-10-21T05:08:05.719657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6230,"log_dups_size":0,"ondisk_log_size":6230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"84'3120","reported_seq":4420,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:08:05.602948+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:08:05.602948+0000","last_peered":"2023-10-21T05:08:05.602948+0000","last_clean":"2023-10-21T05:08:05.602948+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:08:05.602948+0000","last_undegraded":"2023-10-21T05:08:05.602948+0000","last_fullsized":"2023-10-21T05:08:05.602948+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3120,"log_dups_size":0,"ondisk_log_size":3120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"84'6180","reported_seq":8921,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:08:05.719744+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:08:05.719744+0000","last_peered":"2023-10-21T05:08:05.719744+0000","last_clean":"2023-10-21T05:08:05.719744+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:08:05.719744+0000","last_undegraded":"2023-10-21T05:08:05.719744+0000","last_fullsized":"2023-10-21T05:08:05.719744+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6180,"log_dups_size":0,"ondisk_log_size":6180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"84'444141","reported_seq":884808,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:04.200241+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:08:04.200241+0000","last_peered":"2023-10-21T05:08:04.200241+0000","last_clean":"2023-10-21T05:08:04.200241+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:08:04.200241+0000","last_undegraded":"2023-10-21T05:08:04.200241+0000","last_fullsized":"2023-10-21T05:08:04.200241+0000","mapping_epoch":16,"log_start":"83'434115","ondisk_log_start":"83'434115","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":68,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 24s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6735126528,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":442137,"num_read_kb":1738760,"num_write":442429,"num_write_kb":1769716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":208,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:30.895797+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:07:30.895797+0000","last_peered":"2023-10-21T05:07:30.895797+0000","last_clean":"2023-10-21T05:07:30.895797+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:07:30.895797+0000","last_undegraded":"2023-10-21T05:07:30.895797+0000","last_fullsized":"2023-10-21T05:07:30.895797+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"84'448024","reported_seq":894150,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:06.026905+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:08:06.026905+0000","last_peered":"2023-10-21T05:08:06.026905+0000","last_clean":"2023-10-21T05:08:06.026905+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:08:06.026905+0000","last_undegraded":"2023-10-21T05:08:06.026905+0000","last_fullsized":"2023-10-21T05:08:06.026905+0000","mapping_epoch":16,"log_start":"83'438018","ondisk_log_start":"83'438018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":62,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 15s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6846058496,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":447672,"num_read_kb":1760240,"num_write":446242,"num_write_kb":1784968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3588,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:08:01.993465+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:08:01.993465+0000","last_peered":"2023-10-21T05:08:01.993465+0000","last_clean":"2023-10-21T05:08:01.993465+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:08:01.993465+0000","last_undegraded":"2023-10-21T05:08:01.993465+0000","last_fullsized":"2023-10-21T05:08:01.993465+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"84'419073","reported_seq":837267,"reported_epoch":84,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:04.207339+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:08:04.207339+0000","last_peered":"2023-10-21T05:08:04.207339+0000","last_clean":"2023-10-21T05:08:04.207339+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:08:04.207339+0000","last_undegraded":"2023-10-21T05:08:04.207339+0000","last_fullsized":"2023-10-21T05:08:04.207339+0000","mapping_epoch":16,"log_start":"83'409018","ondisk_log_start":"83'409018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":150,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 28s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6346305536,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":419536,"num_read_kb":1649444,"num_write":417433,"num_write_kb":1669732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3524,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:08:01.993641+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:08:01.993641+0000","last_peered":"2023-10-21T05:08:01.993641+0000","last_clean":"2023-10-21T05:08:01.993641+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:08:01.993641+0000","last_undegraded":"2023-10-21T05:08:01.993641+0000","last_fullsized":"2023-10-21T05:08:01.993641+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"84'437345","reported_seq":871346,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:08:06.018490+0000","last_change":"2023-10-21T05:08:05.726457+0000","last_active":"2023-10-21T05:08:06.018490+0000","last_peered":"2023-10-21T05:08:06.018490+0000","last_clean":"2023-10-21T05:08:06.018490+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:08:06.018490+0000","last_undegraded":"2023-10-21T05:08:06.018490+0000","last_fullsized":"2023-10-21T05:08:06.018490+0000","mapping_epoch":16,"log_start":"83'427324","ondisk_log_start":"83'427324","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6660182035,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":435496,"num_read_kb":1712012,"num_write":435654,"num_write_kb":1742610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"84'15238","reported_seq":21078,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:08:05.793172+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:08:05.793172+0000","last_peered":"2023-10-21T05:08:05.793172+0000","last_clean":"2023-10-21T05:08:05.793172+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:08:05.793172+0000","last_undegraded":"2023-10-21T05:08:05.793172+0000","last_fullsized":"2023-10-21T05:08:05.793172+0000","mapping_epoch":13,"log_start":"80'5200","ondisk_log_start":"80'5200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":254,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:28.876030+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:07:28.876030+0000","last_peered":"2023-10-21T05:07:28.876030+0000","last_clean":"2023-10-21T05:07:28.876030+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:07:28.876030+0000","last_undegraded":"2023-10-21T05:07:28.876030+0000","last_fullsized":"2023-10-21T05:07:28.876030+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26587672595,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1744841,"num_read_kb":6860456,"num_write":1741758,"num_write_kb":6967026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14269345792,"data_stored":17163317286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40108,"ondisk_log_size":40108,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":35891,"ondisk_log_size":35891,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739171,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8677016,"kb_used_data":6968208,"kb_used_omap":667,"kb_used_meta":1708132,"kb_avail":85694824,"statfs":{"total":96636764160,"available":87751499776,"internally_reserved":0,"allocated":7135444992,"data_stored":8581941015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":683328,"internal_metadata":1749127872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:07:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.40400000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.51300000000000001,"5min":3.7160000000000002,"15min":4.6139999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.373,"15min":0.41999999999999998},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":0.54900000000000004,"5min":3.7749999999999999,"15min":5.0979999999999999},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Oct 21 05:07:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60199999999999998,"15min":0.72099999999999997},"min":{"1min":0.20499999999999999,"5min":0.192,"15min":0.17499999999999999},"max":{"1min":4.6950000000000003,"5min":5.4199999999999999,"15min":31.763999999999999},"last":0.44},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.50700000000000001,"15min":0.56999999999999995},"min":{"1min":0.26600000000000001,"5min":0.219,"15min":0.20300000000000001},"max":{"1min":0.53100000000000003,"5min":5.2199999999999998,"15min":31.832999999999998},"last":0.497}]}]},{"osd":1,"up_from":8,"seq":34359739168,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5124540,"kb_used_data":3528176,"kb_used_omap":353,"kb_used_meta":1595998,"kb_avail":89247300,"statfs":{"total":96636764160,"available":91389235200,"internally_reserved":0,"allocated":3612852224,"data_stored":4358778729,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1634302167},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,5,5,13,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:07:34 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47299999999999998,"15min":0.53700000000000003},"min":{"1min":0.17699999999999999,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":2.6030000000000002,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.42999999999999999,"15min":0.54100000000000004},"min":{"1min":0.24099999999999999,"5min":0.19800000000000001,"15min":0.17499999999999999},"max":{"1min":0.60099999999999998,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":0.32800000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:07:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.49299999999999999,"15min":0.434},"min":{"1min":0.217,"5min":0.20399999999999999,"15min":0.19800000000000001},"max":{"1min":2.6440000000000001,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.377},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.437,"15min":0.433},"min":{"1min":0.28599999999999998,"5min":0.23300000000000001,"15min":0.22},"max":{"1min":1.0620000000000001,"5min":4.306,"15min":5.3380000000000001},"last":0.28499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739171,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4808004,"kb_used_data":3440124,"kb_used_omap":329,"kb_used_meta":1367542,"kb_avail":89563836,"statfs":{"total":96636764160,"available":91713368064,"internally_reserved":0,"allocated":3522686976,"data_stored":4223601466,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1400363018},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.35099999999999998,"15min":0.57299999999999995},"min":{"1min":0.19500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":55.045999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40999999999999998,"15min":0.626},"min":{"1min":0.30599999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":55.079999999999998},"last":0.41999999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.39500000000000002,"15min":0.438},"min":{"1min":0.254,"5min":0.221,"15min":0.19},"max":{"1min":0.76400000000000001,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41599999999999998,"15min":0.47399999999999998},"min":{"1min":0.28100000000000003,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.67400000000000004,"5min":1.71,"15min":5.4900000000000002},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3521998848,"data_stored":4222935040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3612241920,"data_stored":4358291475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7135105024,"data_stored":8582090771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:08:09.658 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:09.659 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:09.659 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:09.660 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:09.660 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:08:29.661 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:08:30.249 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:08:30.250 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:08:30.277 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2094,"stamp":"2023-10-21T05:08:30.087941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26616126593,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1771900,"num_read_kb":6908569,"num_write":1750502,"num_write_kb":7001251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76380,"ondisk_log_size":76380,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18832552,"kb_used_data":13997200,"kb_used_omap":1351,"kb_used_meta":4833976,"kb_avail":264282968,"statfs":{"total":289910292480,"available":270625759232,"internally_reserved":0,"allocated":14333132800,"data_stored":17226457260,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1383519,"internal_metadata":4949992353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,5,14,21,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15646720,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4469,"num_read_kb":17824,"num_write":4505,"num_write_kb":18020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001980"},"pg_stats":[{"pgid":"2.7","version":"85'12233","reported_seq":16844,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:23.758638+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:08:23.758638+0000","last_peered":"2023-10-21T05:08:23.758638+0000","last_clean":"2023-10-21T05:08:23.758638+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:08:23.758638+0000","last_undegraded":"2023-10-21T05:08:23.758638+0000","last_fullsized":"2023-10-21T05:08:23.758638+0000","mapping_epoch":13,"log_start":"61'2200","ondisk_log_start":"61'2200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"85'6244","reported_seq":9584,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:23.738874+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:08:23.738874+0000","last_peered":"2023-10-21T05:08:23.738874+0000","last_clean":"2023-10-21T05:08:23.738874+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:08:23.738874+0000","last_undegraded":"2023-10-21T05:08:23.738874+0000","last_fullsized":"2023-10-21T05:08:23.738874+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6244,"log_dups_size":0,"ondisk_log_size":6244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'3127","reported_seq":4432,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:23.014932+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:08:23.014932+0000","last_peered":"2023-10-21T05:08:23.014932+0000","last_clean":"2023-10-21T05:08:23.014932+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:08:23.014932+0000","last_undegraded":"2023-10-21T05:08:23.014932+0000","last_fullsized":"2023-10-21T05:08:23.014932+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"85'6194","reported_seq":8944,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:23.734884+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:08:23.734884+0000","last_peered":"2023-10-21T05:08:23.734884+0000","last_clean":"2023-10-21T05:08:23.734884+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:08:23.734884+0000","last_undegraded":"2023-10-21T05:08:23.734884+0000","last_fullsized":"2023-10-21T05:08:23.734884+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6194,"log_dups_size":0,"ondisk_log_size":6194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'446399","reported_seq":889370,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:29.378756+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:08:29.378756+0000","last_peered":"2023-10-21T05:08:29.378756+0000","last_clean":"2023-10-21T05:08:29.378756+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:08:29.378756+0000","last_undegraded":"2023-10-21T05:08:29.378756+0000","last_fullsized":"2023-10-21T05:08:29.378756+0000","mapping_epoch":16,"log_start":"83'436315","ondisk_log_start":"83'436315","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":180,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 49s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6742560768,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":444430,"num_read_kb":1747912,"num_write":444622,"num_write_kb":1778488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":209,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:30.895874+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:07:30.895874+0000","last_peered":"2023-10-21T05:07:30.895874+0000","last_clean":"2023-10-21T05:07:30.895874+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:07:30.895874+0000","last_undegraded":"2023-10-21T05:07:30.895874+0000","last_fullsized":"2023-10-21T05:07:30.895874+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"85'449910","reported_seq":897875,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:26.079133+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:08:26.079133+0000","last_peered":"2023-10-21T05:08:26.079133+0000","last_clean":"2023-10-21T05:08:26.079133+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:08:26.079133+0000","last_undegraded":"2023-10-21T05:08:26.079133+0000","last_fullsized":"2023-10-21T05:08:26.079133+0000","mapping_epoch":16,"log_start":"83'439818","ondisk_log_start":"83'439818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":145,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 35s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6852628480,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":449497,"num_read_kb":1767524,"num_write":448077,"num_write_kb":1792308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3602,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:22.510192+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:08:22.510192+0000","last_peered":"2023-10-21T05:08:22.510192+0000","last_clean":"2023-10-21T05:08:22.510192+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:08:22.510192+0000","last_undegraded":"2023-10-21T05:08:22.510192+0000","last_fullsized":"2023-10-21T05:08:22.510192+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"85'421376","reported_seq":841730,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:29.382902+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:08:29.382902+0000","last_peered":"2023-10-21T05:08:29.382902+0000","last_clean":"2023-10-21T05:08:29.382902+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:08:29.382902+0000","last_undegraded":"2023-10-21T05:08:29.382902+0000","last_fullsized":"2023-10-21T05:08:29.382902+0000","mapping_epoch":16,"log_start":"83'411318","ondisk_log_start":"83'411318","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":249,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 53s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6354014208,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":421680,"num_read_kb":1658008,"num_write":419674,"num_write_kb":1678696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3537,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:22.475670+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:08:22.475670+0000","last_peered":"2023-10-21T05:08:22.475670+0000","last_clean":"2023-10-21T05:08:22.475670+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:08:22.475670+0000","last_undegraded":"2023-10-21T05:08:22.475670+0000","last_fullsized":"2023-10-21T05:08:22.475670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"85'439082","reported_seq":874803,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:26.081824+0000","last_change":"2023-10-21T05:08:23.024683+0000","last_active":"2023-10-21T05:08:26.081824+0000","last_peered":"2023-10-21T05:08:26.081824+0000","last_clean":"2023-10-21T05:08:26.081824+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:08:26.081824+0000","last_undegraded":"2023-10-21T05:08:26.081824+0000","last_fullsized":"2023-10-21T05:08:26.081824+0000","mapping_epoch":16,"log_start":"83'429024","ondisk_log_start":"83'429024","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6666330131,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":437205,"num_read_kb":1718804,"num_write":437391,"num_write_kb":1749558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'15273","reported_seq":21134,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:23.837285+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:08:23.837285+0000","last_peered":"2023-10-21T05:08:23.837285+0000","last_clean":"2023-10-21T05:08:23.837285+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:08:23.837285+0000","last_undegraded":"2023-10-21T05:08:23.837285+0000","last_fullsized":"2023-10-21T05:08:23.837285+0000","mapping_epoch":13,"log_start":"80'5200","ondisk_log_start":"80'5200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":255,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:55.968777+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:07:55.968777+0000","last_peered":"2023-10-21T05:07:55.968777+0000","last_clean":"2023-10-21T05:07:55.968777+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:07:55.968777+0000","last_undegraded":"2023-10-21T05:07:55.968777+0000","last_fullsized":"2023-10-21T05:07:55.968777+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26615533587,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1752812,"num_read_kb":6892248,"num_write":1749764,"num_write_kb":6999050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14331305984,"data_stored":17225277478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40292,"ondisk_log_size":40292,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":35989,"ondisk_log_size":35989,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739175,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8791668,"kb_used_data":6996844,"kb_used_omap":667,"kb_used_meta":1794148,"kb_avail":85580172,"statfs":{"total":96636764160,"available":87634096128,"internally_reserved":0,"allocated":7164768256,"data_stored":8611260093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":683328,"internal_metadata":1837208256},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,5,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:07:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.40400000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.51300000000000001,"5min":3.7160000000000002,"15min":4.6139999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.373,"15min":0.41999999999999998},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":0.54900000000000004,"5min":3.7749999999999999,"15min":5.0979999999999999},"last":0.33700000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:07:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60199999999999998,"15min":0.72099999999999997},"min":{"1min":0.20499999999999999,"5min":0.192,"15min":0.17499999999999999},"max":{"1min":4.6950000000000003,"5min":5.4199999999999999,"15min":31.763999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.50700000000000001,"15min":0.56999999999999995},"min":{"1min":0.26600000000000001,"5min":0.219,"15min":0.20300000000000001},"max":{"1min":0.53100000000000003,"5min":5.2199999999999998,"15min":31.832999999999998},"last":0.45100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739172,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5175036,"kb_used_data":3542580,"kb_used_omap":353,"kb_used_meta":1632094,"kb_avail":89196804,"statfs":{"total":96636764160,"available":91337527296,"internally_reserved":0,"allocated":3627601920,"data_stored":4373524239,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1671264471},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:07:34 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47299999999999998,"15min":0.53700000000000003},"min":{"1min":0.17699999999999999,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":2.6030000000000002,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.42999999999999999,"15min":0.54100000000000004},"min":{"1min":0.24099999999999999,"5min":0.19800000000000001,"15min":0.17499999999999999},"max":{"1min":0.60099999999999998,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:07:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.49299999999999999,"15min":0.434},"min":{"1min":0.217,"5min":0.20399999999999999,"15min":0.19800000000000001},"max":{"1min":2.6440000000000001,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.437,"15min":0.433},"min":{"1min":0.28599999999999998,"5min":0.23300000000000001,"15min":0.22},"max":{"1min":1.0620000000000001,"5min":4.306,"15min":5.3380000000000001},"last":0.379}]}]},{"osd":0,"up_from":8,"seq":34359739176,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4865848,"kb_used_data":3457776,"kb_used_omap":329,"kb_used_meta":1407734,"kb_avail":89505992,"statfs":{"total":96636764160,"available":91654135808,"internally_reserved":0,"allocated":3540762624,"data_stored":4241672928,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1441519626},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,6,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.35099999999999998,"15min":0.57299999999999995},"min":{"1min":0.19500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":55.045999999999999},"last":0.315},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40999999999999998,"15min":0.626},"min":{"1min":0.30599999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":55.079999999999998},"last":0.376}]},{"osd":2,"last update":"Sat Oct 21 05:07:41 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.39500000000000002,"15min":0.438},"min":{"1min":0.254,"5min":0.221,"15min":0.19},"max":{"1min":0.76400000000000001,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41599999999999998,"15min":0.47399999999999998},"min":{"1min":0.28100000000000003,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.67400000000000004,"5min":1.71,"15min":5.4900000000000002},"last":0.52500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3540164608,"data_stored":4241100800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3626893312,"data_stored":4372942867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7164248064,"data_stored":8611233811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:08:30.278 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:30.279 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:30.279 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:30.279 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:30.280 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:08:50.281 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:08:50.722 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:08:50.723 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:08:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2104,"stamp":"2023-10-21T05:08:50.091302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26640743553,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1778985,"num_read_kb":6936773,"num_write":1757545,"num_write_kb":7029423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76323,"ondisk_log_size":76323,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19190700,"kb_used_data":14053452,"kb_used_omap":1351,"kb_used_meta":5135864,"kb_avail":263924820,"statfs":{"total":289910292480,"available":270259015680,"internally_reserved":0,"allocated":14390734848,"data_stored":17284059308,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1383519,"internal_metadata":5259125665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[19,12,7,3,4,8,20,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14884864,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4436,"num_read_kb":17668,"num_write":4313,"num_write_kb":17252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002078"},"pg_stats":[{"pgid":"2.7","version":"85'12269","reported_seq":16896,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:44.686482+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:08:44.686482+0000","last_peered":"2023-10-21T05:08:44.686482+0000","last_clean":"2023-10-21T05:08:44.686482+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:08:44.686482+0000","last_undegraded":"2023-10-21T05:08:44.686482+0000","last_fullsized":"2023-10-21T05:08:44.686482+0000","mapping_epoch":13,"log_start":"61'2200","ondisk_log_start":"61'2200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"85'6262","reported_seq":9610,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:44.655743+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:08:44.655743+0000","last_peered":"2023-10-21T05:08:44.655743+0000","last_clean":"2023-10-21T05:08:44.655743+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:08:44.655743+0000","last_undegraded":"2023-10-21T05:08:44.655743+0000","last_fullsized":"2023-10-21T05:08:44.655743+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'3136","reported_seq":4446,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:42.953037+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:08:42.953037+0000","last_peered":"2023-10-21T05:08:42.953037+0000","last_clean":"2023-10-21T05:08:42.953037+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:08:42.953037+0000","last_undegraded":"2023-10-21T05:08:42.953037+0000","last_fullsized":"2023-10-21T05:08:42.953037+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3136,"log_dups_size":0,"ondisk_log_size":3136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"85'6212","reported_seq":8971,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:44.657509+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:08:44.657509+0000","last_peered":"2023-10-21T05:08:44.657509+0000","last_clean":"2023-10-21T05:08:44.657509+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:08:44.657509+0000","last_undegraded":"2023-10-21T05:08:44.657509+0000","last_fullsized":"2023-10-21T05:08:44.657509+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6212,"log_dups_size":0,"ondisk_log_size":6212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'448276","reported_seq":893056,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:49.569382+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:08:49.569382+0000","last_peered":"2023-10-21T05:08:49.569382+0000","last_clean":"2023-10-21T05:08:49.569382+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:08:49.569382+0000","last_undegraded":"2023-10-21T05:08:49.569382+0000","last_fullsized":"2023-10-21T05:08:49.569382+0000","mapping_epoch":16,"log_start":"83'438215","ondisk_log_start":"83'438215","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":269,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 69s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6748749824,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":446225,"num_read_kb":1755040,"num_write":446441,"num_write_kb":1785764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":210,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:09.762238+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:08:09.762238+0000","last_peered":"2023-10-21T05:08:09.762238+0000","last_clean":"2023-10-21T05:08:09.762238+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:08:09.762238+0000","last_undegraded":"2023-10-21T05:08:09.762238+0000","last_fullsized":"2023-10-21T05:08:09.762238+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"85'451760","reported_seq":901571,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:46.143409+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:08:46.143409+0000","last_peered":"2023-10-21T05:08:46.143409+0000","last_clean":"2023-10-21T05:08:46.143409+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:08:46.143409+0000","last_undegraded":"2023-10-21T05:08:46.143409+0000","last_fullsized":"2023-10-21T05:08:46.143409+0000","mapping_epoch":16,"log_start":"83'441718","ondisk_log_start":"83'441718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":187,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 55s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6859124736,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":451336,"num_read_kb":1774844,"num_write":449901,"num_write_kb":1799604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3614,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:42.954099+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:08:42.954099+0000","last_peered":"2023-10-21T05:08:42.954099+0000","last_clean":"2023-10-21T05:08:42.954099+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:08:42.954099+0000","last_undegraded":"2023-10-21T05:08:42.954099+0000","last_fullsized":"2023-10-21T05:08:42.954099+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"85'423095","reported_seq":845129,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:08:49.563928+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:08:49.563928+0000","last_peered":"2023-10-21T05:08:49.563928+0000","last_clean":"2023-10-21T05:08:49.563928+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:08:49.563928+0000","last_undegraded":"2023-10-21T05:08:49.563928+0000","last_fullsized":"2023-10-21T05:08:49.563928+0000","mapping_epoch":16,"log_start":"83'413018","ondisk_log_start":"83'413018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":395,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 73s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6359601152,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":423348,"num_read_kb":1664652,"num_write":421303,"num_write_kb":1685212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3549,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:42.953577+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:08:42.953577+0000","last_peered":"2023-10-21T05:08:42.953577+0000","last_clean":"2023-10-21T05:08:42.953577+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:08:42.953577+0000","last_undegraded":"2023-10-21T05:08:42.953577+0000","last_fullsized":"2023-10-21T05:08:42.953577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"85'440853","reported_seq":878357,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:46.155428+0000","last_change":"2023-10-21T05:08:45.819023+0000","last_active":"2023-10-21T05:08:46.155428+0000","last_peered":"2023-10-21T05:08:46.155428+0000","last_clean":"2023-10-21T05:08:46.155428+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:08:46.155428+0000","last_undegraded":"2023-10-21T05:08:46.155428+0000","last_fullsized":"2023-10-21T05:08:46.155428+0000","mapping_epoch":16,"log_start":"83'430824","ondisk_log_start":"83'430824","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6672674835,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":438988,"num_read_kb":1725916,"num_write":439162,"num_write_kb":1756642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'15318","reported_seq":21200,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:44.729167+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:08:44.729167+0000","last_peered":"2023-10-21T05:08:44.729167+0000","last_clean":"2023-10-21T05:08:44.729167+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:08:44.729167+0000","last_undegraded":"2023-10-21T05:08:44.729167+0000","last_fullsized":"2023-10-21T05:08:44.729167+0000","mapping_epoch":13,"log_start":"80'5300","ondisk_log_start":"80'5300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":255,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:55.968777+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:07:55.968777+0000","last_peered":"2023-10-21T05:07:55.968777+0000","last_clean":"2023-10-21T05:07:55.968777+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:07:55.968777+0000","last_undegraded":"2023-10-21T05:07:55.968777+0000","last_fullsized":"2023-10-21T05:07:55.968777+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26640150547,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1759897,"num_read_kb":6920452,"num_write":1756807,"num_write_kb":7027222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14389190656,"data_stored":17283162150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40209,"ondisk_log_size":40209,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36015,"ondisk_log_size":36015,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739179,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9000836,"kb_used_data":7024884,"kb_used_omap":667,"kb_used_meta":1975268,"kb_avail":85371004,"statfs":{"total":96636764160,"available":87419908096,"internally_reserved":0,"allocated":7193481216,"data_stored":8639973053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":683328,"internal_metadata":2022675136},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:08:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.41899999999999998,"15min":0.41799999999999998},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3599999999999999,"5min":3.7160000000000002,"15min":4.6139999999999999},"last":3.6600000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.39000000000000001,"15min":0.40799999999999997},"min":{"1min":0.219,"5min":0.185,"15min":0.185},"max":{"1min":1.4510000000000001,"5min":3.7749999999999999,"15min":4.6639999999999997},"last":3.7029999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:08:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.64400000000000002,"15min":0.623},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.17499999999999999},"max":{"1min":1.8700000000000001,"5min":5.4199999999999999,"15min":9.2569999999999997},"last":4.0609999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54800000000000004,"15min":0.47399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":3.109,"5min":5.2199999999999998,"15min":5.2199999999999998},"last":4.274}]}]},{"osd":1,"up_from":8,"seq":34359739176,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5242176,"kb_used_data":3556984,"kb_used_omap":353,"kb_used_meta":1684830,"kb_avail":89129664,"statfs":{"total":96636764160,"available":91268775936,"internally_reserved":0,"allocated":3642351616,"data_stored":4388273935,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1725266135},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[17,10,3,2,0,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:08:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.53300000000000003,"15min":0.54500000000000004},"min":{"1min":0.20899999999999999,"5min":0.17000000000000001,"15min":0.16500000000000001},"max":{"1min":3.3180000000000001,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46400000000000002,"15min":0.54100000000000004},"min":{"1min":0.186,"5min":0.186,"15min":0.17499999999999999},"max":{"1min":3.3559999999999999,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":0.22500000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:08:38 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51400000000000001,"15min":0.44},"min":{"1min":0.219,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":2.1680000000000001,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.308},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.443,"15min":0.42499999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.4590000000000001,"5min":4.306,"15min":5.3380000000000001},"last":0.376}]}]},{"osd":0,"up_from":8,"seq":34359739180,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4947688,"kb_used_data":3471584,"kb_used_omap":329,"kb_used_meta":1475766,"kb_avail":89424152,"statfs":{"total":96636764160,"available":91570331648,"internally_reserved":0,"allocated":3554902016,"data_stored":4255812320,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1511184394},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,3,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.35499999999999998,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.52200000000000002,"5min":0.69999999999999996,"15min":55.045999999999999},"last":0.378},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41199999999999998,"15min":0.627},"min":{"1min":0.26900000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.55100000000000005,"5min":0.80700000000000005,"15min":55.079999999999998},"last":0.42799999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.39400000000000002,"15min":0.43099999999999999},"min":{"1min":0.25700000000000001,"5min":0.221,"15min":0.19},"max":{"1min":0.53600000000000003,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41899999999999998,"15min":0.47499999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.59099999999999997,"5min":1.71,"15min":5.4900000000000002},"last":0.40799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3554291712,"data_stored":4255227904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3641606144,"data_stored":4387655699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7193292800,"data_stored":8640278547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:08:50.758 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:50.758 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:50.759 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:50.759 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:08:50.759 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:09:10.761 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:09:11.278 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:09:11.279 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:09:11.308 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2114,"stamp":"2023-10-21T05:09:10.094582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26665495681,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1786124,"num_read_kb":6965149,"num_write":1764703,"num_write_kb":7058055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76461,"ondisk_log_size":76461,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19248676,"kb_used_data":14110476,"kb_used_omap":1351,"kb_used_meta":5136824,"kb_avail":263866844,"statfs":{"total":289910292480,"available":270199648256,"internally_reserved":0,"allocated":14449127424,"data_stored":17342451884,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1383519,"internal_metadata":5260108705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,8,6,10,22,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15294464,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4470,"num_read_kb":17752,"num_write":4461,"num_write_kb":17844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001960"},"pg_stats":[{"pgid":"2.7","version":"85'12329","reported_seq":16972,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:09:05.105744+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:09:05.105744+0000","last_peered":"2023-10-21T05:09:05.105744+0000","last_clean":"2023-10-21T05:09:05.105744+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:09:05.105744+0000","last_undegraded":"2023-10-21T05:09:05.105744+0000","last_fullsized":"2023-10-21T05:09:05.105744+0000","mapping_epoch":13,"log_start":"61'2300","ondisk_log_start":"61'2300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"85'6292","reported_seq":9649,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:09:05.084859+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:09:05.084859+0000","last_peered":"2023-10-21T05:09:05.084859+0000","last_clean":"2023-10-21T05:09:05.084859+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:09:05.084859+0000","last_undegraded":"2023-10-21T05:09:05.084859+0000","last_fullsized":"2023-10-21T05:09:05.084859+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'3151","reported_seq":4465,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:09:03.434041+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:09:03.434041+0000","last_peered":"2023-10-21T05:09:03.434041+0000","last_clean":"2023-10-21T05:09:03.434041+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:09:03.434041+0000","last_undegraded":"2023-10-21T05:09:03.434041+0000","last_fullsized":"2023-10-21T05:09:03.434041+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3151,"log_dups_size":0,"ondisk_log_size":3151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"85'6242","reported_seq":9009,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:09:05.081987+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:09:05.081987+0000","last_peered":"2023-10-21T05:09:05.081987+0000","last_clean":"2023-10-21T05:09:05.081987+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:09:05.081987+0000","last_undegraded":"2023-10-21T05:09:05.081987+0000","last_fullsized":"2023-10-21T05:09:05.081987+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6242,"log_dups_size":0,"ondisk_log_size":6242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'450120","reported_seq":896722,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:09.635314+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:09:09.635314+0000","last_peered":"2023-10-21T05:09:09.635314+0000","last_clean":"2023-10-21T05:09:09.635314+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:09:09.635314+0000","last_undegraded":"2023-10-21T05:09:09.635314+0000","last_fullsized":"2023-10-21T05:09:09.635314+0000","mapping_epoch":16,"log_start":"84'440115","ondisk_log_start":"84'440115","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":337,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 89s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6755086336,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":448036,"num_read_kb":1762220,"num_write":448246,"num_write_kb":1792984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":210,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:09.762238+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:08:09.762238+0000","last_peered":"2023-10-21T05:08:09.762238+0000","last_clean":"2023-10-21T05:08:09.762238+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:08:09.762238+0000","last_undegraded":"2023-10-21T05:08:09.762238+0000","last_fullsized":"2023-10-21T05:08:09.762238+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"85'453601","reported_seq":905220,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:06.210405+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:09:06.210405+0000","last_peered":"2023-10-21T05:09:06.210405+0000","last_clean":"2023-10-21T05:09:06.210405+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:09:06.210405+0000","last_undegraded":"2023-10-21T05:09:06.210405+0000","last_fullsized":"2023-10-21T05:09:06.210405+0000","mapping_epoch":16,"log_start":"84'443518","ondisk_log_start":"84'443518","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":280,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 75s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6865289216,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":453134,"num_read_kb":1782004,"num_write":451684,"num_write_kb":1806736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3626,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:09:03.432458+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:09:03.432458+0000","last_peered":"2023-10-21T05:09:03.432458+0000","last_clean":"2023-10-21T05:09:03.432458+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:09:03.432458+0000","last_undegraded":"2023-10-21T05:09:03.432458+0000","last_fullsized":"2023-10-21T05:09:03.432458+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"85'424915","reported_seq":848690,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:09.640206+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:09:09.640206+0000","last_peered":"2023-10-21T05:09:09.640206+0000","last_clean":"2023-10-21T05:09:09.640206+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:09:09.640206+0000","last_undegraded":"2023-10-21T05:09:09.640206+0000","last_fullsized":"2023-10-21T05:09:09.640206+0000","mapping_epoch":16,"log_start":"84'414818","ondisk_log_start":"84'414818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":512,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 93s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6365462528,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":425079,"num_read_kb":1671524,"num_write":423050,"num_write_kb":1692200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3561,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:09:03.436053+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:09:03.436053+0000","last_peered":"2023-10-21T05:09:03.436053+0000","last_clean":"2023-10-21T05:09:03.436053+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:09:03.436053+0000","last_undegraded":"2023-10-21T05:09:03.436053+0000","last_fullsized":"2023-10-21T05:09:03.436053+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"85'442676","reported_seq":881992,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:09:06.210614+0000","last_change":"2023-10-21T05:09:00.011662+0000","last_active":"2023-10-21T05:09:06.210614+0000","last_peered":"2023-10-21T05:09:06.210614+0000","last_clean":"2023-10-21T05:09:06.210614+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:09:06.210614+0000","last_undegraded":"2023-10-21T05:09:06.210614+0000","last_fullsized":"2023-10-21T05:09:06.210614+0000","mapping_epoch":16,"log_start":"84'432624","ondisk_log_start":"84'432624","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6679064595,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":440787,"num_read_kb":1733080,"num_write":440985,"num_write_kb":1763934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'15393","reported_seq":21295,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:09:05.153166+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:09:05.153166+0000","last_peered":"2023-10-21T05:09:05.153166+0000","last_clean":"2023-10-21T05:09:05.153166+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:09:05.153166+0000","last_undegraded":"2023-10-21T05:09:05.153166+0000","last_fullsized":"2023-10-21T05:09:05.153166+0000","mapping_epoch":13,"log_start":"80'5300","ondisk_log_start":"80'5300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":255,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-21T05:07:55.968777+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:07:55.968777+0000","last_peered":"2023-10-21T05:07:55.968777+0000","last_clean":"2023-10-21T05:07:55.968777+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:07:55.968777+0000","last_undegraded":"2023-10-21T05:07:55.968777+0000","last_fullsized":"2023-10-21T05:07:55.968777+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26664902675,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1767036,"num_read_kb":6948828,"num_write":1763965,"num_write_kb":7055854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14447755264,"data_stored":17341726758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40237,"ondisk_log_size":40237,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36125,"ondisk_log_size":36125,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739183,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8962088,"kb_used_data":7053216,"kb_used_omap":667,"kb_used_meta":1908196,"kb_avail":85409752,"statfs":{"total":96636764160,"available":87459586048,"internally_reserved":0,"allocated":7222493184,"data_stored":8668985021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":683328,"internal_metadata":1953993408},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:08:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.41899999999999998,"15min":0.41799999999999998},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3599999999999999,"5min":3.7160000000000002,"15min":4.6139999999999999},"last":0.153},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.39000000000000001,"15min":0.40799999999999997},"min":{"1min":0.219,"5min":0.185,"15min":0.185},"max":{"1min":1.4510000000000001,"5min":3.7749999999999999,"15min":4.6639999999999997},"last":0.223}]},{"osd":1,"last update":"Sat Oct 21 05:08:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.64400000000000002,"15min":0.623},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.17499999999999999},"max":{"1min":1.8700000000000001,"5min":5.4199999999999999,"15min":9.2569999999999997},"last":0.318},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54800000000000004,"15min":0.47399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":3.109,"5min":5.2199999999999998,"15min":5.2199999999999998},"last":0.30299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739180,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5306676,"kb_used_data":3571500,"kb_used_omap":353,"kb_used_meta":1734814,"kb_avail":89065164,"statfs":{"total":96636764160,"available":91202727936,"internally_reserved":0,"allocated":3657216000,"data_stored":4403138319,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1776449751},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,4,2,2,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:08:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.53300000000000003,"15min":0.54500000000000004},"min":{"1min":0.20899999999999999,"5min":0.17000000000000001,"15min":0.16500000000000001},"max":{"1min":3.3180000000000001,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46400000000000002,"15min":0.54100000000000004},"min":{"1min":0.186,"5min":0.186,"15min":0.17499999999999999},"max":{"1min":3.3559999999999999,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":0.30499999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:08:38 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51400000000000001,"15min":0.44},"min":{"1min":0.219,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":2.1680000000000001,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.443,"15min":0.42499999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.4590000000000001,"5min":4.306,"15min":5.3380000000000001},"last":0.46000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739184,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4979912,"kb_used_data":3485760,"kb_used_omap":329,"kb_used_meta":1493814,"kb_avail":89391928,"statfs":{"total":96636764160,"available":91537334272,"internally_reserved":0,"allocated":3569418240,"data_stored":4270328544,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1529665546},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,3,2,5,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.35499999999999998,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.52200000000000002,"5min":0.69999999999999996,"15min":55.045999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41199999999999998,"15min":0.627},"min":{"1min":0.26900000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.55100000000000005,"5min":0.80700000000000005,"15min":55.079999999999998},"last":0.314}]},{"osd":2,"last update":"Sat Oct 21 05:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.39400000000000002,"15min":0.43099999999999999},"min":{"1min":0.25700000000000001,"5min":0.221,"15min":0.19},"max":{"1min":0.53600000000000003,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41899999999999998,"15min":0.47499999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.59099999999999997,"5min":1.71,"15min":5.4900000000000002},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3568840704,"data_stored":4269776896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3656368128,"data_stored":4402417683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7222546432,"data_stored":8669532179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:09:11.310 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:11.310 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:11.311 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:11.311 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:11.311 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:09:31.313 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:09:31.770 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:09:31.771 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:09:31.797 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2125,"stamp":"2023-10-21T05:09:30.099233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26689784961,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1793134,"num_read_kb":6993077,"num_write":1771759,"num_write_kb":7086279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76418,"ondisk_log_size":76418,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19484396,"kb_used_data":14167508,"kb_used_omap":1351,"kb_used_meta":5315512,"kb_avail":263631124,"statfs":{"total":289910292480,"available":269958270976,"internally_reserved":0,"allocated":14507528192,"data_stored":17400839965,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1383519,"internal_metadata":5443085217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,4,6,3,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11759616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3449,"num_read_kb":13736,"num_write":3446,"num_write_kb":13784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.320858"},"pg_stats":[{"pgid":"2.7","version":"86'12365","reported_seq":17026,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:26.150042+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:09:26.150042+0000","last_peered":"2023-10-21T05:09:26.150042+0000","last_clean":"2023-10-21T05:09:26.150042+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:09:26.150042+0000","last_undegraded":"2023-10-21T05:09:26.150042+0000","last_fullsized":"2023-10-21T05:09:26.150042+0000","mapping_epoch":13,"log_start":"61'2300","ondisk_log_start":"61'2300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'6310","reported_seq":9676,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:26.139226+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:09:26.139226+0000","last_peered":"2023-10-21T05:09:26.139226+0000","last_clean":"2023-10-21T05:09:26.139226+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:09:26.139226+0000","last_undegraded":"2023-10-21T05:09:26.139226+0000","last_fullsized":"2023-10-21T05:09:26.139226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6310,"log_dups_size":0,"ondisk_log_size":6310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'3160","reported_seq":4479,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:23.912499+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:09:23.912499+0000","last_peered":"2023-10-21T05:09:23.912499+0000","last_clean":"2023-10-21T05:09:23.912499+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:09:23.912499+0000","last_undegraded":"2023-10-21T05:09:23.912499+0000","last_fullsized":"2023-10-21T05:09:23.912499+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3160,"log_dups_size":0,"ondisk_log_size":3160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'6260","reported_seq":9036,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:26.139307+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:09:26.139307+0000","last_peered":"2023-10-21T05:09:26.139307+0000","last_clean":"2023-10-21T05:09:26.139307+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:09:26.139307+0000","last_undegraded":"2023-10-21T05:09:26.139307+0000","last_fullsized":"2023-10-21T05:09:26.139307+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"86'451990","reported_seq":900395,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:29.792157+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:09:29.792157+0000","last_peered":"2023-10-21T05:09:29.792157+0000","last_clean":"2023-10-21T05:09:29.792157+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:09:29.792157+0000","last_undegraded":"2023-10-21T05:09:29.792157+0000","last_fullsized":"2023-10-21T05:09:29.792157+0000","mapping_epoch":16,"log_start":"84'441915","ondisk_log_start":"84'441915","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":391,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 109s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6761467904,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":449834,"num_read_kb":1769396,"num_write":450083,"num_write_kb":1800332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":211,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:39.460915+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:08:39.460915+0000","last_peered":"2023-10-21T05:08:39.460915+0000","last_clean":"2023-10-21T05:08:39.460915+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:08:39.460915+0000","last_undegraded":"2023-10-21T05:08:39.460915+0000","last_fullsized":"2023-10-21T05:08:39.460915+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"86'455499","reported_seq":908939,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:26.373114+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:09:26.373114+0000","last_peered":"2023-10-21T05:09:26.373114+0000","last_clean":"2023-10-21T05:09:26.373114+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:09:26.373114+0000","last_undegraded":"2023-10-21T05:09:26.373114+0000","last_fullsized":"2023-10-21T05:09:26.373114+0000","mapping_epoch":16,"log_start":"84'445418","ondisk_log_start":"84'445418","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":378,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 95s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6871646208,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":454940,"num_read_kb":1789188,"num_write":453518,"num_write_kb":1814072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3639,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:23.951182+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:09:23.951182+0000","last_peered":"2023-10-21T05:09:23.951182+0000","last_clean":"2023-10-21T05:09:23.951182+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:09:23.951182+0000","last_undegraded":"2023-10-21T05:09:23.951182+0000","last_fullsized":"2023-10-21T05:09:23.951182+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"86'426625","reported_seq":852078,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:29.772613+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:09:29.772613+0000","last_peered":"2023-10-21T05:09:29.772613+0000","last_clean":"2023-10-21T05:09:29.772613+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:09:29.772613+0000","last_undegraded":"2023-10-21T05:09:29.772613+0000","last_fullsized":"2023-10-21T05:09:29.772613+0000","mapping_epoch":16,"log_start":"84'416618","ondisk_log_start":"84'416618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":639,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 113s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6370942976,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":426737,"num_read_kb":1678144,"num_write":424682,"num_write_kb":1698728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3574,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:23.960087+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:09:23.960087+0000","last_peered":"2023-10-21T05:09:23.960087+0000","last_clean":"2023-10-21T05:09:23.960087+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:09:23.960087+0000","last_undegraded":"2023-10-21T05:09:23.960087+0000","last_fullsized":"2023-10-21T05:09:23.960087+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"86'444429","reported_seq":885496,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:26.383844+0000","last_change":"2023-10-21T05:09:24.273645+0000","last_active":"2023-10-21T05:09:26.383844+0000","last_peered":"2023-10-21T05:09:26.383844+0000","last_clean":"2023-10-21T05:09:26.383844+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:09:26.383844+0000","last_undegraded":"2023-10-21T05:09:26.383844+0000","last_fullsized":"2023-10-21T05:09:26.383844+0000","mapping_epoch":16,"log_start":"84'434424","ondisk_log_start":"84'434424","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6685134867,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":442535,"num_read_kb":1740028,"num_write":442738,"num_write_kb":1770946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"86'15438","reported_seq":21361,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:26.221573+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:09:26.221573+0000","last_peered":"2023-10-21T05:09:26.221573+0000","last_clean":"2023-10-21T05:09:26.221573+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:09:26.221573+0000","last_undegraded":"2023-10-21T05:09:26.221573+0000","last_fullsized":"2023-10-21T05:09:26.221573+0000","mapping_epoch":13,"log_start":"80'5400","ondisk_log_start":"80'5400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":256,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:09.746378+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:08:09.746378+0000","last_peered":"2023-10-21T05:08:09.746378+0000","last_clean":"2023-10-21T05:08:09.746378+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:08:09.746378+0000","last_undegraded":"2023-10-21T05:08:09.746378+0000","last_fullsized":"2023-10-21T05:08:09.746378+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26689191955,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1774046,"num_read_kb":6976756,"num_write":1771021,"num_write_kb":7084078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14505975808,"data_stored":17399947302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40168,"ondisk_log_size":40168,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36151,"ondisk_log_size":36151,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739187,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9054512,"kb_used_data":7081768,"kb_used_omap":667,"kb_used_meta":1972068,"kb_avail":85317328,"statfs":{"total":96636764160,"available":87364943872,"internally_reserved":0,"allocated":7251730432,"data_stored":8698218040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":683328,"internal_metadata":2019398336},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:08:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.41899999999999998,"15min":0.41799999999999998},"min":{"1min":0.188,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":3.3599999999999999,"5min":3.7160000000000002,"15min":4.6139999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.39000000000000001,"15min":0.40799999999999997},"min":{"1min":0.219,"5min":0.185,"15min":0.185},"max":{"1min":1.4510000000000001,"5min":3.7749999999999999,"15min":4.6639999999999997},"last":0.442}]},{"osd":1,"last update":"Sat Oct 21 05:08:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.64400000000000002,"15min":0.623},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.17499999999999999},"max":{"1min":1.8700000000000001,"5min":5.4199999999999999,"15min":9.2569999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54800000000000004,"15min":0.47399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":3.109,"5min":5.2199999999999998,"15min":5.2199999999999998},"last":0.29099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739184,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5357252,"kb_used_data":3585980,"kb_used_omap":353,"kb_used_meta":1770910,"kb_avail":89014588,"statfs":{"total":96636764160,"available":91150938112,"internally_reserved":0,"allocated":3672043520,"data_stored":4417961610,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1813412055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,2,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:08:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.53300000000000003,"15min":0.54500000000000004},"min":{"1min":0.20899999999999999,"5min":0.17000000000000001,"15min":0.16500000000000001},"max":{"1min":3.3180000000000001,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46400000000000002,"15min":0.54100000000000004},"min":{"1min":0.186,"5min":0.186,"15min":0.17499999999999999},"max":{"1min":3.3559999999999999,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":0.221}]},{"osd":2,"last update":"Sat Oct 21 05:08:38 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51400000000000001,"15min":0.44},"min":{"1min":0.219,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":2.1680000000000001,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.443,"15min":0.42499999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.4590000000000001,"5min":4.306,"15min":5.3380000000000001},"last":0.25600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739188,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072632,"kb_used_data":3499760,"kb_used_omap":329,"kb_used_meta":1572534,"kb_avail":89299208,"statfs":{"total":96636764160,"available":91442388992,"internally_reserved":0,"allocated":3583754240,"data_stored":4284660315,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1610274826},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,3,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.35499999999999998,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.52200000000000002,"5min":0.69999999999999996,"15min":55.045999999999999},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41199999999999998,"15min":0.627},"min":{"1min":0.26900000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.55100000000000005,"5min":0.80700000000000005,"15min":55.079999999999998},"last":0.26200000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:08:45 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.39400000000000002,"15min":0.43099999999999999},"min":{"1min":0.25700000000000001,"5min":0.221,"15min":0.19},"max":{"1min":0.53600000000000003,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41899999999999998,"15min":0.47499999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.59099999999999997,"5min":1.71,"15min":5.4900000000000002},"last":0.28999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3583041536,"data_stored":4283977728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3671072768,"data_stored":4417122323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7251861504,"data_stored":8698847251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:09:31.799 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:31.800 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:31.800 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:31.800 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:31.801 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:09:33.502 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:09:33.495+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:09:33.534 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:09:33.531+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:09:33.534 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:09:33.531+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:09:33.535 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:09:33.531+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:09:51.802 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:09:51.812 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:09:51.791+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:09:51.813 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:09:51.803+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:09:51.819 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:09:51.803+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:09:52.344 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:09:52.345 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:09:52.395 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2136,"stamp":"2023-10-21T05:09:52.102965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26717809793,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1801225,"num_read_kb":7025285,"num_write":1779827,"num_write_kb":7118551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76480,"ondisk_log_size":76480,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19749232,"kb_used_data":14234840,"kb_used_omap":1351,"kb_used_meta":5513016,"kb_avail":263366288,"statfs":{"total":289910292480,"available":269687078912,"internally_reserved":0,"allocated":14576476160,"data_stored":17469787933,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1383519,"internal_metadata":5645329313},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,2,1,2,13,12,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15495168,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4489,"num_read_kb":17880,"num_write":4464,"num_write_kb":17856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001999"},"pg_stats":[{"pgid":"2.7","version":"86'12426","reported_seq":17107,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:49.516407+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:09:49.516407+0000","last_peered":"2023-10-21T05:09:49.516407+0000","last_clean":"2023-10-21T05:09:49.516407+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:09:49.516407+0000","last_undegraded":"2023-10-21T05:09:49.516407+0000","last_fullsized":"2023-10-21T05:09:49.516407+0000","mapping_epoch":13,"log_start":"61'2400","ondisk_log_start":"61'2400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'6341","reported_seq":9717,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:49.515980+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:09:49.515980+0000","last_peered":"2023-10-21T05:09:49.515980+0000","last_clean":"2023-10-21T05:09:49.515980+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:09:49.515980+0000","last_undegraded":"2023-10-21T05:09:49.515980+0000","last_fullsized":"2023-10-21T05:09:49.515980+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6341,"log_dups_size":0,"ondisk_log_size":6341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'3176","reported_seq":4500,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:47.992566+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:09:47.992566+0000","last_peered":"2023-10-21T05:09:47.992566+0000","last_clean":"2023-10-21T05:09:47.992566+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:09:47.992566+0000","last_undegraded":"2023-10-21T05:09:47.992566+0000","last_fullsized":"2023-10-21T05:09:47.992566+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3176,"log_dups_size":0,"ondisk_log_size":3176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'6291","reported_seq":9077,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:49.516333+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:09:49.516333+0000","last_peered":"2023-10-21T05:09:49.516333+0000","last_clean":"2023-10-21T05:09:49.516333+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:09:49.516333+0000","last_undegraded":"2023-10-21T05:09:49.516333+0000","last_fullsized":"2023-10-21T05:09:49.516333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6291,"log_dups_size":0,"ondisk_log_size":6291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"86'453881","reported_seq":904107,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:49.945063+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:09:49.945063+0000","last_peered":"2023-10-21T05:09:49.945063+0000","last_clean":"2023-10-21T05:09:49.945063+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:09:49.945063+0000","last_undegraded":"2023-10-21T05:09:49.945063+0000","last_fullsized":"2023-10-21T05:09:49.945063+0000","mapping_epoch":16,"log_start":"84'443815","ondisk_log_start":"84'443815","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":488,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 130s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6767747072,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":451641,"num_read_kb":1776592,"num_write":451916,"num_write_kb":1807664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":211,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:39.460915+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:08:39.460915+0000","last_peered":"2023-10-21T05:08:39.460915+0000","last_clean":"2023-10-21T05:08:39.460915+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:08:39.460915+0000","last_undegraded":"2023-10-21T05:08:39.460915+0000","last_fullsized":"2023-10-21T05:08:39.460915+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"86'457857","reported_seq":913597,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:51.509600+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:09:51.509600+0000","last_peered":"2023-10-21T05:09:51.509600+0000","last_clean":"2023-10-21T05:09:51.509600+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:09:51.509600+0000","last_undegraded":"2023-10-21T05:09:51.509600+0000","last_fullsized":"2023-10-21T05:09:51.509600+0000","mapping_epoch":16,"log_start":"84'447818","ondisk_log_start":"84'447818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":464,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 120s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6879780864,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":457229,"num_read_kb":1798300,"num_write":455828,"num_write_kb":1823312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3654,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:49.524717+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:09:49.524717+0000","last_peered":"2023-10-21T05:09:49.524717+0000","last_clean":"2023-10-21T05:09:49.524717+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:09:49.524717+0000","last_undegraded":"2023-10-21T05:09:49.524717+0000","last_fullsized":"2023-10-21T05:09:49.524717+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"86'428413","reported_seq":855593,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:09:49.923650+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:09:49.923650+0000","last_peered":"2023-10-21T05:09:49.923650+0000","last_clean":"2023-10-21T05:09:49.923650+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:09:49.923650+0000","last_undegraded":"2023-10-21T05:09:49.923650+0000","last_fullsized":"2023-10-21T05:09:49.923650+0000","mapping_epoch":16,"log_start":"84'418318","ondisk_log_start":"84'418318","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":762,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 134s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6376755200,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":428455,"num_read_kb":1684988,"num_write":426396,"num_write_kb":1705584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3589,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:49.517254+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:09:49.517254+0000","last_peered":"2023-10-21T05:09:49.517254+0000","last_clean":"2023-10-21T05:09:49.517254+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:09:49.517254+0000","last_undegraded":"2023-10-21T05:09:49.517254+0000","last_fullsized":"2023-10-21T05:09:49.517254+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"86'446640","reported_seq":889994,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:51.509946+0000","last_change":"2023-10-21T05:09:48.332953+0000","last_active":"2023-10-21T05:09:51.509946+0000","last_peered":"2023-10-21T05:09:51.509946+0000","last_clean":"2023-10-21T05:09:51.509946+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:09:51.509946+0000","last_undegraded":"2023-10-21T05:09:51.509946+0000","last_fullsized":"2023-10-21T05:09:51.509946+0000","mapping_epoch":16,"log_start":"84'436624","ondisk_log_start":"84'436624","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6692933651,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":444812,"num_read_kb":1749084,"num_write":444949,"num_write_kb":1779790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"86'15513","reported_seq":21461,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:49.516897+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:09:49.516897+0000","last_peered":"2023-10-21T05:09:49.516897+0000","last_clean":"2023-10-21T05:09:49.516897+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:09:49.516897+0000","last_undegraded":"2023-10-21T05:09:49.516897+0000","last_fullsized":"2023-10-21T05:09:49.516897+0000","mapping_epoch":13,"log_start":"80'5500","ondisk_log_start":"80'5500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":256,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:09.746378+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:08:09.746378+0000","last_peered":"2023-10-21T05:08:09.746378+0000","last_clean":"2023-10-21T05:08:09.746378+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:08:09.746378+0000","last_undegraded":"2023-10-21T05:08:09.746378+0000","last_fullsized":"2023-10-21T05:08:09.746378+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26717216787,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1782137,"num_read_kb":7008964,"num_write":1779089,"num_write_kb":7116350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14575394816,"data_stored":17469366310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40216,"ondisk_log_size":40216,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36165,"ondisk_log_size":36165,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739192,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9208596,"kb_used_data":7117132,"kb_used_omap":667,"kb_used_meta":2090788,"kb_avail":85163244,"statfs":{"total":96636764160,"available":87207161856,"internally_reserved":0,"allocated":7287943168,"data_stored":8734430776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":683328,"internal_metadata":2140967616},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.432,"15min":0.41199999999999998},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":3.6600000000000001,"5min":3.6600000000000001,"15min":3.7160000000000002},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.437,"15min":0.41399999999999998},"min":{"1min":0.20799999999999999,"5min":0.185,"15min":0.185},"max":{"1min":4.319,"5min":4.319,"15min":4.319},"last":0.27600000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:09:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.219,"5min":0.20200000000000001,"15min":0.17499999999999999},"max":{"1min":4.0609999999999999,"5min":5.4199999999999999,"15min":9.2569999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55500000000000005,"15min":0.48699999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20300000000000001},"max":{"1min":4.274,"5min":5.2199999999999998,"15min":5.2199999999999998},"last":0.42599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739189,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5440156,"kb_used_data":3603988,"kb_used_omap":353,"kb_used_meta":1835806,"kb_avail":88931684,"statfs":{"total":96636764160,"available":91066044416,"internally_reserved":0,"allocated":3690483712,"data_stored":4436401802,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1879865559},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,0,1,1,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57699999999999996,"15min":0.53700000000000003},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.16900000000000001},"max":{"1min":2.8919999999999999,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":4.5599999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.501,"15min":0.53100000000000003},"min":{"1min":0.217,"5min":0.186,"15min":0.17499999999999999},"max":{"1min":2.9180000000000001,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":4.6109999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.50900000000000001,"15min":0.44},"min":{"1min":0.20899999999999999,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":0.52500000000000002,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.441,"15min":0.41999999999999998},"min":{"1min":0.25600000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.54200000000000004,"5min":4.306,"15min":5.3380000000000001},"last":0.30599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739192,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5100480,"kb_used_data":3513720,"kb_used_omap":329,"kb_used_meta":1586422,"kb_avail":89271360,"statfs":{"total":96636764160,"available":91413872640,"internally_reserved":0,"allocated":3598049280,"data_stored":4298955355,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1624496138},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,9,4,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:09:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35999999999999999,"15min":0.56999999999999995},"min":{"1min":0.20599999999999999,"5min":0.19500000000000001,"15min":0.17199999999999999},"max":{"1min":0.67100000000000004,"5min":0.69999999999999996,"15min":55.045999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41499999999999998,"15min":0.623},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.217},"max":{"1min":0.70199999999999996,"5min":0.80700000000000005,"15min":55.079999999999998},"last":0.52400000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:09:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40100000000000002,"15min":0.42699999999999999},"min":{"1min":0.23799999999999999,"5min":0.22700000000000001,"15min":0.19},"max":{"1min":0.73299999999999998,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42899999999999999,"15min":0.45500000000000002},"min":{"1min":0.251,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.63700000000000001,"5min":1.71,"15min":4.0609999999999999},"last":0.56599999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3597565952,"data_stored":4298502144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3689578496,"data_stored":4435628051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7288250368,"data_stored":8735236115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:09:52.397 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:52.397 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:52.398 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:52.398 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:09:52.398 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:10:12.400 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:10:13.008 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:10:13.009 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:10:13.045 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2146,"stamp":"2023-10-21T05:10:12.110174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26742328449,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1808325,"num_read_kb":7053581,"num_write":1786948,"num_write_kb":7147035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76653,"ondisk_log_size":76653,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19272736,"kb_used_data":14291844,"kb_used_omap":1352,"kb_used_meta":4979511,"kb_avail":263842784,"statfs":{"total":289910292480,"available":270175010816,"internally_reserved":0,"allocated":14634848256,"data_stored":17528160029,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1385199,"internal_metadata":5099019537},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,0,9,12,16,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15384576,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4473,"num_read_kb":17820,"num_write":4449,"num_write_kb":17796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002132"},"pg_stats":[{"pgid":"2.7","version":"86'12479","reported_seq":17176,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:10:10.005501+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:10:10.005501+0000","last_peered":"2023-10-21T05:10:10.005501+0000","last_clean":"2023-10-21T05:10:10.005501+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:10:10.005501+0000","last_undegraded":"2023-10-21T05:10:10.005501+0000","last_fullsized":"2023-10-21T05:10:10.005501+0000","mapping_epoch":13,"log_start":"61'2400","ondisk_log_start":"61'2400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'6368","reported_seq":9752,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:10:09.992163+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:10:09.992163+0000","last_peered":"2023-10-21T05:10:09.992163+0000","last_clean":"2023-10-21T05:10:09.992163+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:10:09.992163+0000","last_undegraded":"2023-10-21T05:10:09.992163+0000","last_fullsized":"2023-10-21T05:10:09.992163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6368,"log_dups_size":0,"ondisk_log_size":6368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'3190","reported_seq":4518,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:10:09.718811+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:10:09.718811+0000","last_peered":"2023-10-21T05:10:09.718811+0000","last_clean":"2023-10-21T05:10:09.718811+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:10:09.718811+0000","last_undegraded":"2023-10-21T05:10:09.718811+0000","last_fullsized":"2023-10-21T05:10:09.718811+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3190,"log_dups_size":0,"ondisk_log_size":3190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'6317","reported_seq":9111,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:10:09.992209+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:10:09.992209+0000","last_peered":"2023-10-21T05:10:09.992209+0000","last_clean":"2023-10-21T05:10:09.992209+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:10:09.992209+0000","last_undegraded":"2023-10-21T05:10:09.992209+0000","last_fullsized":"2023-10-21T05:10:09.992209+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6317,"log_dups_size":0,"ondisk_log_size":6317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"86'455723","reported_seq":907738,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:10.044304+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:10:10.044304+0000","last_peered":"2023-10-21T05:10:10.044304+0000","last_clean":"2023-10-21T05:10:10.044304+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:10:10.044304+0000","last_undegraded":"2023-10-21T05:10:10.044304+0000","last_fullsized":"2023-10-21T05:10:10.044304+0000","mapping_epoch":16,"log_start":"85'445715","ondisk_log_start":"85'445715","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":564,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 150s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6773760000,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":453418,"num_read_kb":1783676,"num_write":453707,"num_write_kb":1814828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":211,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-21T05:08:39.460915+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:08:39.460915+0000","last_peered":"2023-10-21T05:08:39.460915+0000","last_clean":"2023-10-21T05:08:39.460915+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:08:39.460915+0000","last_undegraded":"2023-10-21T05:08:39.460915+0000","last_fullsized":"2023-10-21T05:08:39.460915+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"86'459781","reported_seq":917381,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:11.598006+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:10:11.598006+0000","last_peered":"2023-10-21T05:10:11.598006+0000","last_clean":"2023-10-21T05:10:11.598006+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:10:11.598006+0000","last_undegraded":"2023-10-21T05:10:11.598006+0000","last_fullsized":"2023-10-21T05:10:11.598006+0000","mapping_epoch":16,"log_start":"85'449718","ondisk_log_start":"85'449718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":526,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 140s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6886379520,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":459076,"num_read_kb":1805664,"num_write":457714,"num_write_kb":1830856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3666,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:10:10.064286+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:10:10.064286+0000","last_peered":"2023-10-21T05:10:10.064286+0000","last_clean":"2023-10-21T05:10:10.064286+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:10:10.064286+0000","last_undegraded":"2023-10-21T05:10:10.064286+0000","last_fullsized":"2023-10-21T05:10:10.064286+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"86'430210","reported_seq":859112,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:10.041473+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:10:10.041473+0000","last_peered":"2023-10-21T05:10:10.041473+0000","last_clean":"2023-10-21T05:10:10.041473+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:10:10.041473+0000","last_undegraded":"2023-10-21T05:10:10.041473+0000","last_fullsized":"2023-10-21T05:10:10.041473+0000","mapping_epoch":16,"log_start":"85'420118","ondisk_log_start":"85'420118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":892,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 154s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6382641152,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":430167,"num_read_kb":1691808,"num_write":428116,"num_write_kb":1712464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3602,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:10:10.009189+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:10:10.009189+0000","last_peered":"2023-10-21T05:10:10.009189+0000","last_clean":"2023-10-21T05:10:10.009189+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:10:10.009189+0000","last_undegraded":"2023-10-21T05:10:10.009189+0000","last_fullsized":"2023-10-21T05:10:10.009189+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"86'448364","reported_seq":893491,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:10:11.589206+0000","last_change":"2023-10-21T05:10:08.488237+0000","last_active":"2023-10-21T05:10:11.589206+0000","last_peered":"2023-10-21T05:10:11.589206+0000","last_clean":"2023-10-21T05:10:11.589206+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:10:11.589206+0000","last_undegraded":"2023-10-21T05:10:11.589206+0000","last_fullsized":"2023-10-21T05:10:11.589206+0000","mapping_epoch":16,"log_start":"85'438324","ondisk_log_start":"85'438324","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6698954771,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":446576,"num_read_kb":1756112,"num_write":446673,"num_write_kb":1786686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"86'15579","reported_seq":21547,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:10:10.007475+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:10:10.007475+0000","last_peered":"2023-10-21T05:10:10.007475+0000","last_clean":"2023-10-21T05:10:10.007475+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:10:10.007475+0000","last_undegraded":"2023-10-21T05:10:10.007475+0000","last_fullsized":"2023-10-21T05:10:10.007475+0000","mapping_epoch":13,"log_start":"80'5500","ondisk_log_start":"80'5500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":257,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:19.788317+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:09:19.788317+0000","last_peered":"2023-10-21T05:09:19.788317+0000","last_clean":"2023-10-21T05:09:19.788317+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:09:19.788317+0000","last_undegraded":"2023-10-21T05:09:19.788317+0000","last_fullsized":"2023-10-21T05:09:19.788317+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26741735443,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1789237,"num_read_kb":7037260,"num_write":1786210,"num_write_kb":7144834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14634053632,"data_stored":17528025126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40203,"ondisk_log_size":40203,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36351,"ondisk_log_size":36351,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739196,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8628948,"kb_used_data":7145676,"kb_used_omap":668,"kb_used_meta":1482595,"kb_avail":85742892,"statfs":{"total":96636764160,"available":87800721408,"internally_reserved":0,"allocated":7317172224,"data_stored":8763659832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":685008,"internal_metadata":1518177328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.432,"15min":0.41199999999999998},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":3.6600000000000001,"5min":3.6600000000000001,"15min":3.7160000000000002},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.437,"15min":0.41399999999999998},"min":{"1min":0.20799999999999999,"5min":0.185,"15min":0.185},"max":{"1min":4.319,"5min":4.319,"15min":4.319},"last":0.27100000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:09:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.219,"5min":0.20200000000000001,"15min":0.17499999999999999},"max":{"1min":4.0609999999999999,"5min":5.4199999999999999,"15min":9.2569999999999997},"last":0.252},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55500000000000005,"15min":0.48699999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20300000000000001},"max":{"1min":4.274,"5min":5.2199999999999998,"15min":5.2199999999999998},"last":0.47599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739193,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5486560,"kb_used_data":3618456,"kb_used_omap":353,"kb_used_meta":1867742,"kb_avail":88885280,"statfs":{"total":96636764160,"available":91018526720,"internally_reserved":0,"allocated":3705298944,"data_stored":4451217034,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1912568023},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57699999999999996,"15min":0.53700000000000003},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.16900000000000001},"max":{"1min":2.8919999999999999,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.501,"15min":0.53100000000000003},"min":{"1min":0.217,"5min":0.186,"15min":0.17499999999999999},"max":{"1min":2.9180000000000001,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":0.374}]},{"osd":2,"last update":"Sat Oct 21 05:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.50900000000000001,"15min":0.44},"min":{"1min":0.20899999999999999,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":0.52500000000000002,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.441,"15min":0.41999999999999998},"min":{"1min":0.25600000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.54200000000000004,"5min":4.306,"15min":5.3380000000000001},"last":0.40799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739196,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5157228,"kb_used_data":3527712,"kb_used_omap":329,"kb_used_meta":1629174,"kb_avail":89214612,"statfs":{"total":96636764160,"available":91355762688,"internally_reserved":0,"allocated":3612377088,"data_stored":4313283163,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1668274186},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,0,2,1,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:09:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35999999999999999,"15min":0.56999999999999995},"min":{"1min":0.20599999999999999,"5min":0.19500000000000001,"15min":0.17199999999999999},"max":{"1min":0.67100000000000004,"5min":0.69999999999999996,"15min":55.045999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41499999999999998,"15min":0.623},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.217},"max":{"1min":0.70199999999999996,"5min":0.80700000000000005,"15min":55.079999999999998},"last":0.32500000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:09:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40100000000000002,"15min":0.42699999999999999},"min":{"1min":0.23799999999999999,"5min":0.22700000000000001,"15min":0.19},"max":{"1min":0.73299999999999998,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42899999999999999,"15min":0.45500000000000002},"min":{"1min":0.251,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.63700000000000001,"5min":1.71,"15min":4.0609999999999999},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3611955200,"data_stored":4312891392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3704393728,"data_stored":4450443283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7317704704,"data_stored":8764690451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:10:13.047 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:13.047 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:13.048 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:13.048 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:13.048 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:10:13.565 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-21T05:10:13.599 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:10:14.221 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:10:14.252 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:10:14.792 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-21T05:10:14.819 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:10:15.292 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:10:15.323 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:10:35.324 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:10:35.891 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:10:35.893 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:10:35.939 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2158,"stamp":"2023-10-21T05:10:34.114077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26766195841,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1815424,"num_read_kb":7081877,"num_write":1793900,"num_write_kb":7174843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76700,"ondisk_log_size":76700,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19484308,"kb_used_data":14348220,"kb_used_omap":1352,"kb_used_meta":5134711,"kb_avail":263631212,"statfs":{"total":289910292480,"available":269958361088,"internally_reserved":0,"allocated":14692577280,"data_stored":17585880509,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1385199,"internal_metadata":5257944337},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,3,15,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12165120,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3747,"num_read_kb":14928,"num_write":3521,"num_write_kb":14084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001785"},"pg_stats":[{"pgid":"2.7","version":"87'12539","reported_seq":17253,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:31.777416+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:10:31.777416+0000","last_peered":"2023-10-21T05:10:31.777416+0000","last_clean":"2023-10-21T05:10:31.777416+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:10:31.777416+0000","last_undegraded":"2023-10-21T05:10:31.777416+0000","last_fullsized":"2023-10-21T05:10:31.777416+0000","mapping_epoch":13,"log_start":"61'2500","ondisk_log_start":"61'2500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"87'6398","reported_seq":9791,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:31.763723+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:10:31.763723+0000","last_peered":"2023-10-21T05:10:31.763723+0000","last_clean":"2023-10-21T05:10:31.763723+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:10:31.763723+0000","last_undegraded":"2023-10-21T05:10:31.763723+0000","last_fullsized":"2023-10-21T05:10:31.763723+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6398,"log_dups_size":0,"ondisk_log_size":6398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"87'3205","reported_seq":4538,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:30.476394+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:10:30.476394+0000","last_peered":"2023-10-21T05:10:30.476394+0000","last_clean":"2023-10-21T05:10:30.476394+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:10:30.476394+0000","last_undegraded":"2023-10-21T05:10:30.476394+0000","last_fullsized":"2023-10-21T05:10:30.476394+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3205,"log_dups_size":0,"ondisk_log_size":3205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"87'6347","reported_seq":9150,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:31.755670+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:10:31.755670+0000","last_peered":"2023-10-21T05:10:31.755670+0000","last_clean":"2023-10-21T05:10:31.755670+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:10:31.755670+0000","last_undegraded":"2023-10-21T05:10:31.755670+0000","last_fullsized":"2023-10-21T05:10:31.755670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6347,"log_dups_size":0,"ondisk_log_size":6347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"87'457537","reported_seq":911364,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:30.221187+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:10:30.221187+0000","last_peered":"2023-10-21T05:10:30.221187+0000","last_clean":"2023-10-21T05:10:30.221187+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:10:30.221187+0000","last_undegraded":"2023-10-21T05:10:30.221187+0000","last_fullsized":"2023-10-21T05:10:30.221187+0000","mapping_epoch":16,"log_start":"85'447515","ondisk_log_start":"85'447515","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":640,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 170s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6779760640,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":455214,"num_read_kb":1790840,"num_write":455471,"num_write_kb":1821884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":212,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:19.793554+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:09:19.793554+0000","last_peered":"2023-10-21T05:09:19.793554+0000","last_clean":"2023-10-21T05:09:19.793554+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:09:19.793554+0000","last_undegraded":"2023-10-21T05:09:19.793554+0000","last_fullsized":"2023-10-21T05:09:19.793554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"87'461648","reported_seq":921194,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:31.818093+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:10:31.818093+0000","last_peered":"2023-10-21T05:10:31.818093+0000","last_clean":"2023-10-21T05:10:31.818093+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:10:31.818093+0000","last_undegraded":"2023-10-21T05:10:31.818093+0000","last_fullsized":"2023-10-21T05:10:31.818093+0000","mapping_epoch":16,"log_start":"85'451618","ondisk_log_start":"85'451618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":635,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 161s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6892691456,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":461012,"num_read_kb":1813380,"num_write":459511,"num_write_kb":1838044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3680,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:30.479448+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:10:30.479448+0000","last_peered":"2023-10-21T05:10:30.479448+0000","last_clean":"2023-10-21T05:10:30.479448+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:10:30.479448+0000","last_undegraded":"2023-10-21T05:10:30.479448+0000","last_fullsized":"2023-10-21T05:10:30.479448+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"87'431916","reported_seq":862478,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:30.172102+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:10:30.172102+0000","last_peered":"2023-10-21T05:10:30.172102+0000","last_clean":"2023-10-21T05:10:30.172102+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:10:30.172102+0000","last_undegraded":"2023-10-21T05:10:30.172102+0000","last_fullsized":"2023-10-21T05:10:30.172102+0000","mapping_epoch":16,"log_start":"85'421818","ondisk_log_start":"85'421818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1002,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 174s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6388121600,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":431810,"num_read_kb":1698352,"num_write":429757,"num_write_kb":1719028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3615,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:30.483360+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:10:30.483360+0000","last_peered":"2023-10-21T05:10:30.483360+0000","last_clean":"2023-10-21T05:10:30.483360+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:10:30.483360+0000","last_undegraded":"2023-10-21T05:10:30.483360+0000","last_fullsized":"2023-10-21T05:10:30.483360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"87'450114","reported_seq":896971,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:31.808454+0000","last_change":"2023-10-21T05:10:27.428027+0000","last_active":"2023-10-21T05:10:31.808454+0000","last_peered":"2023-10-21T05:10:31.808454+0000","last_clean":"2023-10-21T05:10:31.808454+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:10:31.808454+0000","last_undegraded":"2023-10-21T05:10:31.808454+0000","last_fullsized":"2023-10-21T05:10:31.808454+0000","mapping_epoch":16,"log_start":"85'440024","ondisk_log_start":"85'440024","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6705029139,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":448300,"num_read_kb":1762984,"num_write":448423,"num_write_kb":1793686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"87'15654","reported_seq":21640,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:31.810440+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:10:31.810440+0000","last_peered":"2023-10-21T05:10:31.810440+0000","last_clean":"2023-10-21T05:10:31.810440+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:10:31.810440+0000","last_undegraded":"2023-10-21T05:10:31.810440+0000","last_fullsized":"2023-10-21T05:10:31.810440+0000","mapping_epoch":13,"log_start":"80'5600","ondisk_log_start":"80'5600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":258,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:56.544723+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:09:56.544723+0000","last_peered":"2023-10-21T05:09:56.544723+0000","last_clean":"2023-10-21T05:09:56.544723+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:09:56.544723+0000","last_undegraded":"2023-10-21T05:09:56.544723+0000","last_fullsized":"2023-10-21T05:09:56.544723+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26765602835,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1796336,"num_read_kb":7065556,"num_write":1793162,"num_write_kb":7172642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14690775040,"data_stored":17584746534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40240,"ondisk_log_size":40240,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36361,"ondisk_log_size":36361,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739200,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8739576,"kb_used_data":7174384,"kb_used_omap":668,"kb_used_meta":1564515,"kb_avail":85632264,"statfs":{"total":96636764160,"available":87687438336,"internally_reserved":0,"allocated":7346569216,"data_stored":8793052552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":685008,"internal_metadata":1602063408},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.432,"15min":0.41199999999999998},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":3.6600000000000001,"5min":3.6600000000000001,"15min":3.7160000000000002},"last":2.8090000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.437,"15min":0.41399999999999998},"min":{"1min":0.20799999999999999,"5min":0.185,"15min":0.185},"max":{"1min":4.319,"5min":4.319,"15min":4.319},"last":2.7370000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:09:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.219,"5min":0.20200000000000001,"15min":0.17499999999999999},"max":{"1min":4.0609999999999999,"5min":5.4199999999999999,"15min":9.2569999999999997},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55500000000000005,"15min":0.48699999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20300000000000001},"max":{"1min":4.274,"5min":5.2199999999999998,"15min":5.2199999999999998},"last":0.38500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739197,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5518732,"kb_used_data":3632580,"kb_used_omap":353,"kb_used_meta":1885790,"kb_avail":88853108,"statfs":{"total":96636764160,"available":90985582592,"internally_reserved":0,"allocated":3719761920,"data_stored":4465675738,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1931049175},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57699999999999996,"15min":0.53700000000000003},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.16900000000000001},"max":{"1min":2.8919999999999999,"5min":4.0090000000000003,"15min":6.8289999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.501,"15min":0.53100000000000003},"min":{"1min":0.217,"5min":0.186,"15min":0.17499999999999999},"max":{"1min":2.9180000000000001,"5min":3.4769999999999999,"15min":6.8460000000000001},"last":0.28100000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.50900000000000001,"15min":0.44},"min":{"1min":0.20899999999999999,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":0.52500000000000002,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.441,"15min":0.41999999999999998},"min":{"1min":0.25600000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.54200000000000004,"5min":4.306,"15min":5.3380000000000001},"last":0.39000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739200,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5226000,"kb_used_data":3541256,"kb_used_omap":329,"kb_used_meta":1684406,"kb_avail":89145840,"statfs":{"total":96636764160,"available":91285340160,"internally_reserved":0,"allocated":3626246144,"data_stored":4327152219,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1724831754},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,11,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:09:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35999999999999999,"15min":0.56999999999999995},"min":{"1min":0.20599999999999999,"5min":0.19500000000000001,"15min":0.17199999999999999},"max":{"1min":0.67100000000000004,"5min":0.69999999999999996,"15min":55.045999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41499999999999998,"15min":0.623},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.217},"max":{"1min":0.70199999999999996,"5min":0.80700000000000005,"15min":55.079999999999998},"last":0.33500000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:09:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40100000000000002,"15min":0.42699999999999999},"min":{"1min":0.23799999999999999,"5min":0.22700000000000001,"15min":0.19},"max":{"1min":0.73299999999999998,"5min":0.79700000000000004,"15min":4.1349999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.42899999999999999,"15min":0.45500000000000002},"min":{"1min":0.251,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.63700000000000001,"5min":1.71,"15min":4.0609999999999999},"last":0.308}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3625893888,"data_stored":4326830080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3718893568,"data_stored":4464943123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7345987584,"data_stored":8792973331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:10:35.940 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:35.941 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:35.941 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:35.942 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:35.942 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:10:55.944 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:10:56.453 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:10:56.455 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:10:56.501 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2169,"stamp":"2023-10-21T05:10:56.119868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26793966721,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1823439,"num_read_kb":7113825,"num_write":1801917,"num_write_kb":7206911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76735,"ondisk_log_size":76735,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19689520,"kb_used_data":14408084,"kb_used_omap":1352,"kb_used_meta":5280055,"kb_avail":263426000,"statfs":{"total":289910292480,"available":269748224000,"internally_reserved":0,"allocated":14753878016,"data_stored":17647176973,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1385199,"internal_metadata":5406776593},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,4,6,13,16,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15417344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4423,"num_read_kb":17628,"num_write":4456,"num_write_kb":17824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004070"},"pg_stats":[{"pgid":"2.7","version":"87'12587","reported_seq":17317,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.466277+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:10:51.466277+0000","last_peered":"2023-10-21T05:10:51.466277+0000","last_clean":"2023-10-21T05:10:51.466277+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:10:51.466277+0000","last_undegraded":"2023-10-21T05:10:51.466277+0000","last_fullsized":"2023-10-21T05:10:51.466277+0000","mapping_epoch":13,"log_start":"61'2500","ondisk_log_start":"61'2500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"87'6422","reported_seq":9823,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.439918+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:10:51.439918+0000","last_peered":"2023-10-21T05:10:51.439918+0000","last_clean":"2023-10-21T05:10:51.439918+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:10:51.439918+0000","last_undegraded":"2023-10-21T05:10:51.439918+0000","last_fullsized":"2023-10-21T05:10:51.439918+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"87'3217","reported_seq":4555,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:50.956406+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:10:50.956406+0000","last_peered":"2023-10-21T05:10:50.956406+0000","last_clean":"2023-10-21T05:10:50.956406+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:10:50.956406+0000","last_undegraded":"2023-10-21T05:10:50.956406+0000","last_fullsized":"2023-10-21T05:10:50.956406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3217,"log_dups_size":0,"ondisk_log_size":3217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"87'6371","reported_seq":9183,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.439826+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:10:51.439826+0000","last_peered":"2023-10-21T05:10:51.439826+0000","last_clean":"2023-10-21T05:10:51.439826+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:10:51.439826+0000","last_undegraded":"2023-10-21T05:10:51.439826+0000","last_fullsized":"2023-10-21T05:10:51.439826+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"87'459797","reported_seq":915880,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:55.428865+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:10:55.428865+0000","last_peered":"2023-10-21T05:10:55.428865+0000","last_clean":"2023-10-21T05:10:55.428865+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:10:55.428865+0000","last_undegraded":"2023-10-21T05:10:55.428865+0000","last_fullsized":"2023-10-21T05:10:55.428865+0000","mapping_epoch":16,"log_start":"85'449715","ondisk_log_start":"85'449715","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":734,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 195s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6787313664,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":457456,"num_read_kb":1799780,"num_write":457672,"num_write_kb":1830688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":213,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:29.798506+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:10:29.798506+0000","last_peered":"2023-10-21T05:10:29.798506+0000","last_clean":"2023-10-21T05:10:29.798506+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:10:29.798506+0000","last_undegraded":"2023-10-21T05:10:29.798506+0000","last_fullsized":"2023-10-21T05:10:29.798506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"87'463508","reported_seq":924918,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:51.924121+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:10:51.924121+0000","last_peered":"2023-10-21T05:10:51.924121+0000","last_clean":"2023-10-21T05:10:51.924121+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:10:51.924121+0000","last_undegraded":"2023-10-21T05:10:51.924121+0000","last_fullsized":"2023-10-21T05:10:51.924121+0000","mapping_epoch":16,"log_start":"85'453418","ondisk_log_start":"85'453418","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":700,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 181s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6899306496,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":462862,"num_read_kb":1820748,"num_write":461332,"num_write_kb":1845328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3692,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:50.959996+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:10:50.959996+0000","last_peered":"2023-10-21T05:10:50.959996+0000","last_clean":"2023-10-21T05:10:50.959996+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:10:50.959996+0000","last_undegraded":"2023-10-21T05:10:50.959996+0000","last_fullsized":"2023-10-21T05:10:50.959996+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"87'434145","reported_seq":866903,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:10:55.418272+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:10:55.418272+0000","last_peered":"2023-10-21T05:10:55.418272+0000","last_clean":"2023-10-21T05:10:55.418272+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:10:55.418272+0000","last_undegraded":"2023-10-21T05:10:55.418272+0000","last_fullsized":"2023-10-21T05:10:55.418272+0000","mapping_epoch":16,"log_start":"85'424118","ondisk_log_start":"85'424118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1088,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 199s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6395387904,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":433987,"num_read_kb":1707036,"num_write":431934,"num_write_kb":1727736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3627,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:50.956955+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:10:50.956955+0000","last_peered":"2023-10-21T05:10:50.956955+0000","last_clean":"2023-10-21T05:10:50.956955+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:10:50.956955+0000","last_undegraded":"2023-10-21T05:10:50.956955+0000","last_fullsized":"2023-10-21T05:10:50.956955+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"87'451932","reported_seq":900540,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.895134+0000","last_change":"2023-10-21T05:10:50.698894+0000","last_active":"2023-10-21T05:10:51.895134+0000","last_peered":"2023-10-21T05:10:51.895134+0000","last_clean":"2023-10-21T05:10:51.895134+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:10:51.895134+0000","last_undegraded":"2023-10-21T05:10:51.895134+0000","last_fullsized":"2023-10-21T05:10:51.895134+0000","mapping_epoch":16,"log_start":"85'441924","ondisk_log_start":"85'441924","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6711365651,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":450046,"num_read_kb":1769940,"num_write":450241,"num_write_kb":1800958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"87'15714","reported_seq":21724,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:51.487129+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:10:51.487129+0000","last_peered":"2023-10-21T05:10:51.487129+0000","last_clean":"2023-10-21T05:10:51.487129+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:10:51.487129+0000","last_undegraded":"2023-10-21T05:10:51.487129+0000","last_fullsized":"2023-10-21T05:10:51.487129+0000","mapping_epoch":13,"log_start":"80'5700","ondisk_log_start":"80'5700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":258,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:56.544723+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:09:56.544723+0000","last_peered":"2023-10-21T05:09:56.544723+0000","last_clean":"2023-10-21T05:09:56.544723+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:09:56.544723+0000","last_undegraded":"2023-10-21T05:09:56.544723+0000","last_fullsized":"2023-10-21T05:09:56.544723+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26793373715,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1804351,"num_read_kb":7097504,"num_write":1801179,"num_write_kb":7204710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14752755712,"data_stored":17646727206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40207,"ondisk_log_size":40207,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36429,"ondisk_log_size":36429,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739204,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8828188,"kb_used_data":7202256,"kb_used_omap":668,"kb_used_meta":1625251,"kb_avail":85543652,"statfs":{"total":96636764160,"available":87596699648,"internally_reserved":0,"allocated":7375110144,"data_stored":8821593480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":685008,"internal_metadata":1664257072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.48299999999999998,"15min":0.43099999999999999},"min":{"1min":0.20799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":2.8090000000000002,"5min":3.6600000000000001,"15min":3.7160000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.48699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26900000000000002,"5min":0.185,"15min":0.185},"max":{"1min":2.7370000000000001,"5min":4.319,"15min":4.319},"last":0.28000000000000003}]},{"osd":1,"last update":"Sat Oct 21 05:10:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.59899999999999998,"15min":0.61699999999999999},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.17499999999999999},"max":{"1min":7.8449999999999998,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.315},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.497,"15min":0.49299999999999999},"min":{"1min":0.26500000000000001,"5min":0.20999999999999999,"15min":0.20300000000000001},"max":{"1min":1.175,"5min":4.274,"15min":5.2199999999999998},"last":0.33300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739201,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5594268,"kb_used_data":3647124,"kb_used_omap":353,"kb_used_meta":1946782,"kb_avail":88777572,"statfs":{"total":96636764160,"available":90908233728,"internally_reserved":0,"allocated":3734654976,"data_stored":4480568794,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1993504983},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,6,12,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.59799999999999998,"15min":0.55300000000000005},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":6.8289999999999997},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.26000000000000001,"5min":0.186,"15min":0.17499999999999999},"max":{"1min":4.6109999999999998,"5min":4.6109999999999998,"15min":6.8460000000000001},"last":0.499}]},{"osd":2,"last update":"Sat Oct 21 05:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.46700000000000003,"15min":0.443},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.66400000000000003,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.437,"15min":0.42499999999999999},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77900000000000003,"5min":4.306,"15min":5.3380000000000001},"last":0.46700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739205,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5267064,"kb_used_data":3558704,"kb_used_omap":329,"kb_used_meta":1708022,"kb_avail":89104776,"statfs":{"total":96636764160,"available":91243290624,"internally_reserved":0,"allocated":3644112896,"data_stored":4345014699,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1749014538},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.374,"15min":0.57799999999999996},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.17199999999999999},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":55.045999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.42899999999999999,"15min":0.63100000000000001},"min":{"1min":0.32500000000000001,"5min":0.26200000000000001,"15min":0.217},"max":{"1min":1.778,"5min":1.778,"15min":55.079999999999998},"last":0.76700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:10:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.42599999999999999,"15min":0.442},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.19},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":4.1349999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.497,"15min":0.48499999999999999},"min":{"1min":0.26700000000000002,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":4.3319999999999999,"5min":4.3319999999999999,"15min":4.3319999999999999},"last":0.84699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3643830272,"data_stored":4344766464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3733798912,"data_stored":4479848467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7375126528,"data_stored":8822112275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:10:56.503 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:56.504 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:56.504 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:56.505 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:10:56.505 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:11:16.506 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:17.028 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:11:17.030 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:11:17.083 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2179,"stamp":"2023-10-21T05:11:16.123919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26818899073,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1830562,"num_read_kb":7142161,"num_write":1809060,"num_write_kb":7235483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76851,"ondisk_log_size":76851,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19925216,"kb_used_data":14464712,"kb_used_omap":1352,"kb_used_meta":5459127,"kb_avail":263190304,"statfs":{"total":289910292480,"available":269506871296,"internally_reserved":0,"allocated":14811865088,"data_stored":17705164045,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1385199,"internal_metadata":5590146321},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,11,5,15,15,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15572992,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4372,"num_read_kb":17380,"num_write":4490,"num_write_kb":17960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002658"},"pg_stats":[{"pgid":"2.7","version":"87'12611","reported_seq":17357,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:11.499506+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:11:11.499506+0000","last_peered":"2023-10-21T05:11:11.499506+0000","last_clean":"2023-10-21T05:11:11.499506+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:11:11.499506+0000","last_undegraded":"2023-10-21T05:11:11.499506+0000","last_fullsized":"2023-10-21T05:11:11.499506+0000","mapping_epoch":13,"log_start":"68'2600","ondisk_log_start":"68'2600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"87'6434","reported_seq":9844,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:11.480984+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:11:11.480984+0000","last_peered":"2023-10-21T05:11:11.480984+0000","last_clean":"2023-10-21T05:11:11.480984+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:11:11.480984+0000","last_undegraded":"2023-10-21T05:11:11.480984+0000","last_fullsized":"2023-10-21T05:11:11.480984+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6434,"log_dups_size":0,"ondisk_log_size":6434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"87'3223","reported_seq":4565,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:11.433010+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:11:11.433010+0000","last_peered":"2023-10-21T05:11:11.433010+0000","last_clean":"2023-10-21T05:11:11.433010+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:11:11.433010+0000","last_undegraded":"2023-10-21T05:11:11.433010+0000","last_fullsized":"2023-10-21T05:11:11.433010+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3223,"log_dups_size":0,"ondisk_log_size":3223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"87'6383","reported_seq":9203,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:11.482825+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:11:11.482825+0000","last_peered":"2023-10-21T05:11:11.482825+0000","last_clean":"2023-10-21T05:11:11.482825+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:11:11.482825+0000","last_undegraded":"2023-10-21T05:11:11.482825+0000","last_fullsized":"2023-10-21T05:11:11.482825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6383,"log_dups_size":0,"ondisk_log_size":6383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"87'461692","reported_seq":919573,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:15.525362+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:11:15.525362+0000","last_peered":"2023-10-21T05:11:15.525362+0000","last_clean":"2023-10-21T05:11:15.525362+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:11:15.525362+0000","last_undegraded":"2023-10-21T05:11:15.525362+0000","last_fullsized":"2023-10-21T05:11:15.525362+0000","mapping_epoch":16,"log_start":"86'451615","ondisk_log_start":"86'451615","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":806,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 215s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6793764864,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":459243,"num_read_kb":1806876,"num_write":459523,"num_write_kb":1838092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":213,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:29.798506+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:10:29.798506+0000","last_peered":"2023-10-21T05:10:29.798506+0000","last_clean":"2023-10-21T05:10:29.798506+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:10:29.798506+0000","last_undegraded":"2023-10-21T05:10:29.798506+0000","last_fullsized":"2023-10-21T05:10:29.798506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"87'465413","reported_seq":928700,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:11.974556+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:11:11.974556+0000","last_peered":"2023-10-21T05:11:11.974556+0000","last_clean":"2023-10-21T05:11:11.974556+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:11:11.974556+0000","last_undegraded":"2023-10-21T05:11:11.974556+0000","last_fullsized":"2023-10-21T05:11:11.974556+0000","mapping_epoch":16,"log_start":"86'455318","ondisk_log_start":"86'455318","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":794,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 201s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6905847808,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":464728,"num_read_kb":1828152,"num_write":463180,"num_write_kb":1852720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3704,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:11.433788+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:11:11.433788+0000","last_peered":"2023-10-21T05:11:11.433788+0000","last_clean":"2023-10-21T05:11:11.433788+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:11:11.433788+0000","last_undegraded":"2023-10-21T05:11:11.433788+0000","last_fullsized":"2023-10-21T05:11:11.433788+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"87'435896","reported_seq":870363,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:15.549055+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:11:15.549055+0000","last_peered":"2023-10-21T05:11:15.549055+0000","last_clean":"2023-10-21T05:11:15.549055+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:11:15.549055+0000","last_undegraded":"2023-10-21T05:11:15.549055+0000","last_fullsized":"2023-10-21T05:11:15.549055+0000","mapping_epoch":16,"log_start":"85'425818","ondisk_log_start":"85'425818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1236,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 219s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6401007616,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":435680,"num_read_kb":1713780,"num_write":433597,"num_write_kb":1734388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3639,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:11.442215+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:11:11.442215+0000","last_peered":"2023-10-21T05:11:11.442215+0000","last_clean":"2023-10-21T05:11:11.442215+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:11:11.442215+0000","last_undegraded":"2023-10-21T05:11:11.442215+0000","last_fullsized":"2023-10-21T05:11:11.442215+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"87'453713","reported_seq":904106,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:12.045954+0000","last_change":"2023-10-21T05:11:03.696392+0000","last_active":"2023-10-21T05:11:12.045954+0000","last_peered":"2023-10-21T05:11:12.045954+0000","last_clean":"2023-10-21T05:11:12.045954+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:11:12.045954+0000","last_undegraded":"2023-10-21T05:11:12.045954+0000","last_fullsized":"2023-10-21T05:11:12.045954+0000","mapping_epoch":16,"log_start":"85'443624","ondisk_log_start":"85'443624","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6717685779,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":451823,"num_read_kb":1777032,"num_write":452022,"num_write_kb":1808082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"87'15744","reported_seq":21774,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:11.529546+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:11:11.529546+0000","last_peered":"2023-10-21T05:11:11.529546+0000","last_clean":"2023-10-21T05:11:11.529546+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:11:11.529546+0000","last_undegraded":"2023-10-21T05:11:11.529546+0000","last_fullsized":"2023-10-21T05:11:11.529546+0000","mapping_epoch":13,"log_start":"80'5700","ondisk_log_start":"80'5700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":258,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:56.544723+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:09:56.544723+0000","last_peered":"2023-10-21T05:09:56.544723+0000","last_clean":"2023-10-21T05:09:56.544723+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:09:56.544723+0000","last_undegraded":"2023-10-21T05:09:56.544723+0000","last_fullsized":"2023-10-21T05:09:56.544723+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26818306067,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1811474,"num_read_kb":7125840,"num_write":1808322,"num_write_kb":7233282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14811107328,"data_stored":17705078822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40339,"ondisk_log_size":40339,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36413,"ondisk_log_size":36413,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739208,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8936820,"kb_used_data":7230380,"kb_used_omap":668,"kb_used_meta":1705763,"kb_avail":85435020,"statfs":{"total":96636764160,"available":87485460480,"internally_reserved":0,"allocated":7403909120,"data_stored":8850392456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":685008,"internal_metadata":1746701360},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.48299999999999998,"15min":0.43099999999999999},"min":{"1min":0.20799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":2.8090000000000002,"5min":3.6600000000000001,"15min":3.7160000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.48699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26900000000000002,"5min":0.185,"15min":0.185},"max":{"1min":2.7370000000000001,"5min":4.319,"15min":4.319},"last":0.23400000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:10:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.59899999999999998,"15min":0.61699999999999999},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.17499999999999999},"max":{"1min":7.8449999999999998,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.497,"15min":0.49299999999999999},"min":{"1min":0.26500000000000001,"5min":0.20999999999999999,"15min":0.20300000000000001},"max":{"1min":1.175,"5min":4.274,"15min":5.2199999999999998},"last":0.499}]}]},{"osd":1,"up_from":8,"seq":34359739205,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5658764,"kb_used_data":3661636,"kb_used_omap":353,"kb_used_meta":1996766,"kb_avail":88713076,"statfs":{"total":96636764160,"available":90842189824,"internally_reserved":0,"allocated":3749515264,"data_stored":4495429082,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":2044688599},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,6,2,5,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.59799999999999998,"15min":0.55300000000000005},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":6.8289999999999997},"last":0.19},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.26000000000000001,"5min":0.186,"15min":0.17499999999999999},"max":{"1min":4.6109999999999998,"5min":4.6109999999999998,"15min":6.8460000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.46700000000000003,"15min":0.443},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.66400000000000003,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.437,"15min":0.42499999999999999},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77900000000000003,"5min":4.306,"15min":5.3380000000000001},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359739209,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5329632,"kb_used_data":3572696,"kb_used_omap":329,"kb_used_meta":1756598,"kb_avail":89042208,"statfs":{"total":96636764160,"available":91179220992,"internally_reserved":0,"allocated":3658440704,"data_stored":4359342507,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1798756362},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,7,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.374,"15min":0.57799999999999996},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.17199999999999999},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":55.045999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.42899999999999999,"15min":0.63100000000000001},"min":{"1min":0.32500000000000001,"5min":0.26200000000000001,"15min":0.217},"max":{"1min":1.778,"5min":1.778,"15min":55.079999999999998},"last":0.39900000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:10:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.42599999999999999,"15min":0.442},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.19},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":4.1349999999999998},"last":0.373},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.497,"15min":0.48499999999999999},"min":{"1min":0.26700000000000002,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":4.3319999999999999,"5min":4.3319999999999999,"15min":4.3319999999999999},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3658223616,"data_stored":4359159808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3748687872,"data_stored":4494737427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7404195840,"data_stored":8851181587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:17.085 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:17.085 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:17.085 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:17.086 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:17.086 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:11:37.088 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:37.646 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:11:37.647 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:11:37.685 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2189,"stamp":"2023-10-21T05:11:36.127405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26843651201,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1837784,"num_read_kb":7170905,"num_write":1816146,"num_write_kb":7263827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76786,"ondisk_log_size":76786,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19479616,"kb_used_data":14521640,"kb_used_omap":1352,"kb_used_meta":4956599,"kb_avail":263635904,"statfs":{"total":289910292480,"available":269963165696,"internally_reserved":0,"allocated":14870159360,"data_stored":17763458317,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1385199,"internal_metadata":5075557649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,4,12,8,13,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15720448,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4452,"num_read_kb":17696,"num_write":4467,"num_write_kb":17868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002099"},"pg_stats":[{"pgid":"2.7","version":"87'12647","reported_seq":17410,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:31.942062+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:11:31.942062+0000","last_peered":"2023-10-21T05:11:31.942062+0000","last_clean":"2023-10-21T05:11:31.942062+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:11:31.942062+0000","last_undegraded":"2023-10-21T05:11:31.942062+0000","last_fullsized":"2023-10-21T05:11:31.942062+0000","mapping_epoch":13,"log_start":"68'2600","ondisk_log_start":"68'2600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"87'6452","reported_seq":9870,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:31.919824+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:11:31.919824+0000","last_peered":"2023-10-21T05:11:31.919824+0000","last_clean":"2023-10-21T05:11:31.919824+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:11:31.919824+0000","last_undegraded":"2023-10-21T05:11:31.919824+0000","last_fullsized":"2023-10-21T05:11:31.919824+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6452,"log_dups_size":0,"ondisk_log_size":6452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"87'3232","reported_seq":4578,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:31.911523+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:11:31.911523+0000","last_peered":"2023-10-21T05:11:31.911523+0000","last_clean":"2023-10-21T05:11:31.911523+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:11:31.911523+0000","last_undegraded":"2023-10-21T05:11:31.911523+0000","last_fullsized":"2023-10-21T05:11:31.911523+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3232,"log_dups_size":0,"ondisk_log_size":3232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"87'6401","reported_seq":9229,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:31.919808+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:11:31.919808+0000","last_peered":"2023-10-21T05:11:31.919808+0000","last_clean":"2023-10-21T05:11:31.919808+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:11:31.919808+0000","last_undegraded":"2023-10-21T05:11:31.919808+0000","last_fullsized":"2023-10-21T05:11:31.919808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6401,"log_dups_size":0,"ondisk_log_size":6401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"87'463524","reported_seq":923219,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:35.749662+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:11:35.749662+0000","last_peered":"2023-10-21T05:11:35.749662+0000","last_clean":"2023-10-21T05:11:35.749662+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:11:35.749662+0000","last_undegraded":"2023-10-21T05:11:35.749662+0000","last_fullsized":"2023-10-21T05:11:35.749662+0000","mapping_epoch":16,"log_start":"86'453515","ondisk_log_start":"86'453515","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":882,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 235s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6800093184,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":461047,"num_read_kb":1814052,"num_write":461309,"num_write_kb":1845236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":213,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:29.798506+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:10:29.798506+0000","last_peered":"2023-10-21T05:10:29.798506+0000","last_clean":"2023-10-21T05:10:29.798506+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:10:29.798506+0000","last_undegraded":"2023-10-21T05:10:29.798506+0000","last_fullsized":"2023-10-21T05:10:29.798506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"87'467296","reported_seq":932398,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:32.162783+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:11:32.162783+0000","last_peered":"2023-10-21T05:11:32.162783+0000","last_clean":"2023-10-21T05:11:32.162783+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:11:32.162783+0000","last_undegraded":"2023-10-21T05:11:32.162783+0000","last_fullsized":"2023-10-21T05:11:32.162783+0000","mapping_epoch":16,"log_start":"86'457218","ondisk_log_start":"86'457218","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":860,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 221s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6912434176,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":466535,"num_read_kb":1835356,"num_write":465020,"num_write_kb":1860080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3716,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:31.921851+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:11:31.921851+0000","last_peered":"2023-10-21T05:11:31.921851+0000","last_clean":"2023-10-21T05:11:31.921851+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:11:31.921851+0000","last_undegraded":"2023-10-21T05:11:31.921851+0000","last_fullsized":"2023-10-21T05:11:31.921851+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"87'437659","reported_seq":873865,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:35.736451+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:11:35.736451+0000","last_peered":"2023-10-21T05:11:35.736451+0000","last_clean":"2023-10-21T05:11:35.736451+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:11:35.736451+0000","last_undegraded":"2023-10-21T05:11:35.736451+0000","last_fullsized":"2023-10-21T05:11:35.736451+0000","mapping_epoch":16,"log_start":"86'427618","ondisk_log_start":"86'427618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1302,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 239s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6406877184,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":437411,"num_read_kb":1720656,"num_write":435326,"num_write_kb":1741304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3651,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:32.058733+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:11:32.058733+0000","last_peered":"2023-10-21T05:11:32.058733+0000","last_clean":"2023-10-21T05:11:32.058733+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:11:32.058733+0000","last_undegraded":"2023-10-21T05:11:32.058733+0000","last_fullsized":"2023-10-21T05:11:32.058733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"87'455444","reported_seq":907725,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:32.179394+0000","last_change":"2023-10-21T05:11:29.864464+0000","last_active":"2023-10-21T05:11:32.179394+0000","last_peered":"2023-10-21T05:11:32.179394+0000","last_clean":"2023-10-21T05:11:32.179394+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:11:32.179394+0000","last_undegraded":"2023-10-21T05:11:32.179394+0000","last_fullsized":"2023-10-21T05:11:32.179394+0000","mapping_epoch":16,"log_start":"86'445424","ondisk_log_start":"86'445424","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6723653651,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":453703,"num_read_kb":1784520,"num_write":453753,"num_write_kb":1815006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"87'15789","reported_seq":21839,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:11:32.028714+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:11:32.028714+0000","last_peered":"2023-10-21T05:11:32.028714+0000","last_clean":"2023-10-21T05:11:32.028714+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:11:32.028714+0000","last_undegraded":"2023-10-21T05:11:32.028714+0000","last_fullsized":"2023-10-21T05:11:32.028714+0000","mapping_epoch":13,"log_start":"80'5700","ondisk_log_start":"80'5700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":258,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-21T05:09:56.544723+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:09:56.544723+0000","last_peered":"2023-10-21T05:09:56.544723+0000","last_clean":"2023-10-21T05:09:56.544723+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:09:56.544723+0000","last_undegraded":"2023-10-21T05:09:56.544723+0000","last_fullsized":"2023-10-21T05:09:56.544723+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26843058195,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1818696,"num_read_kb":7154584,"num_write":1815408,"num_write_kb":7261626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14869041152,"data_stored":17763012646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40148,"ondisk_log_size":40148,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36539,"ondisk_log_size":36539,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739212,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9032892,"kb_used_data":7258420,"kb_used_omap":668,"kb_used_meta":1773795,"kb_avail":85338948,"statfs":{"total":96636764160,"available":87387082752,"internally_reserved":0,"allocated":7432622080,"data_stored":8879105416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":685008,"internal_metadata":1816366128},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.48299999999999998,"15min":0.43099999999999999},"min":{"1min":0.20799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":2.8090000000000002,"5min":3.6600000000000001,"15min":3.7160000000000002},"last":4.7699999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.48699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26900000000000002,"5min":0.185,"15min":0.185},"max":{"1min":2.7370000000000001,"5min":4.319,"15min":4.319},"last":0.26100000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:10:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.59899999999999998,"15min":0.61699999999999999},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.17499999999999999},"max":{"1min":7.8449999999999998,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.497,"15min":0.49299999999999999},"min":{"1min":0.26500000000000001,"5min":0.20999999999999999,"15min":0.20300000000000001},"max":{"1min":1.175,"5min":4.274,"15min":5.2199999999999998},"last":4.8170000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739209,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5048376,"kb_used_data":3675952,"kb_used_omap":353,"kb_used_meta":1372062,"kb_avail":89323464,"statfs":{"total":96636764160,"available":91467227136,"internally_reserved":0,"allocated":3764174848,"data_stored":4510088666,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1404991703},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,5,3,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.59799999999999998,"15min":0.55300000000000005},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":6.8289999999999997},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.26000000000000001,"5min":0.186,"15min":0.17499999999999999},"max":{"1min":4.6109999999999998,"5min":4.6109999999999998,"15min":6.8460000000000001},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.46700000000000003,"15min":0.443},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.66400000000000003,"5min":4.2729999999999997,"15min":5.0220000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.437,"15min":0.42499999999999999},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77900000000000003,"5min":4.306,"15min":5.3380000000000001},"last":0.24099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739213,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5398348,"kb_used_data":3587268,"kb_used_omap":329,"kb_used_meta":1810742,"kb_avail":88973492,"statfs":{"total":96636764160,"available":91108855808,"internally_reserved":0,"allocated":3673362432,"data_stored":4374264235,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1854199818},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,0,7,3,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:10:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.374,"15min":0.57799999999999996},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.17199999999999999},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":55.045999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.42899999999999999,"15min":0.63100000000000001},"min":{"1min":0.32500000000000001,"5min":0.26200000000000001,"15min":0.217},"max":{"1min":1.778,"5min":1.778,"15min":55.079999999999998},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:10:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.42599999999999999,"15min":0.442},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.19},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":4.1349999999999998},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.497,"15min":0.48499999999999999},"min":{"1min":0.26700000000000002,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":4.3319999999999999,"5min":4.3319999999999999,"15min":4.3319999999999999},"last":0.27500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3672616960,"data_stored":4373553152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3763298304,"data_stored":4509347859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7433125888,"data_stored":8880111635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:37.686 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:37.687 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:37.687 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:37.688 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:37.688 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:11:57.689 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:11:58.271 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:11:58.272 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:11:58.334 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2201,"stamp":"2023-10-21T05:11:58.131058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26871577729,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1845931,"num_read_kb":7203393,"num_write":1824170,"num_write_kb":7295923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76919,"ondisk_log_size":76919,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19761600,"kb_used_data":14590248,"kb_used_omap":1352,"kb_used_meta":5169975,"kb_avail":263353920,"statfs":{"total":289910292480,"available":269674414080,"internally_reserved":0,"allocated":14940413952,"data_stored":17833699964,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1385199,"internal_metadata":5294054673},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,10,3,10,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15695872,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4557,"num_read_kb":18180,"num_write":4471,"num_write_kb":17884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001850"},"pg_stats":[{"pgid":"2.7","version":"88'12711","reported_seq":17491,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:57.064780+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:11:57.064780+0000","last_peered":"2023-10-21T05:11:57.064780+0000","last_clean":"2023-10-21T05:11:57.064780+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:11:57.064780+0000","last_undegraded":"2023-10-21T05:11:57.064780+0000","last_fullsized":"2023-10-21T05:11:57.064780+0000","mapping_epoch":13,"log_start":"68'2700","ondisk_log_start":"68'2700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'6484","reported_seq":9911,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:57.043567+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:11:57.043567+0000","last_peered":"2023-10-21T05:11:57.043567+0000","last_clean":"2023-10-21T05:11:57.043567+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:11:57.043567+0000","last_undegraded":"2023-10-21T05:11:57.043567+0000","last_fullsized":"2023-10-21T05:11:57.043567+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6484,"log_dups_size":0,"ondisk_log_size":6484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"88'3248","reported_seq":4599,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:53.234337+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:11:53.234337+0000","last_peered":"2023-10-21T05:11:53.234337+0000","last_clean":"2023-10-21T05:11:53.234337+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:11:53.234337+0000","last_undegraded":"2023-10-21T05:11:53.234337+0000","last_fullsized":"2023-10-21T05:11:53.234337+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3248,"log_dups_size":0,"ondisk_log_size":3248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"88'6433","reported_seq":9270,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:57.043771+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:11:57.043771+0000","last_peered":"2023-10-21T05:11:57.043771+0000","last_clean":"2023-10-21T05:11:57.043771+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:11:57.043771+0000","last_undegraded":"2023-10-21T05:11:57.043771+0000","last_fullsized":"2023-10-21T05:11:57.043771+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6433,"log_dups_size":0,"ondisk_log_size":6433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"88'465393","reported_seq":926900,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:55.875044+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:11:55.875044+0000","last_peered":"2023-10-21T05:11:55.875044+0000","last_clean":"2023-10-21T05:11:55.875044+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:11:55.875044+0000","last_undegraded":"2023-10-21T05:11:55.875044+0000","last_fullsized":"2023-10-21T05:11:55.875044+0000","mapping_epoch":16,"log_start":"86'455315","ondisk_log_start":"86'455315","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":950,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 256s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6806388736,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":462851,"num_read_kb":1821248,"num_write":463137,"num_write_kb":1852548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":214,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:45.363521+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:10:45.363521+0000","last_peered":"2023-10-21T05:10:45.363521+0000","last_clean":"2023-10-21T05:10:45.363521+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:10:45.363521+0000","last_undegraded":"2023-10-21T05:10:45.363521+0000","last_fullsized":"2023-10-21T05:10:45.363521+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"88'469568","reported_seq":936978,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:57.256378+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:11:57.256378+0000","last_peered":"2023-10-21T05:11:57.256378+0000","last_clean":"2023-10-21T05:11:57.256378+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:11:57.256378+0000","last_undegraded":"2023-10-21T05:11:57.256378+0000","last_fullsized":"2023-10-21T05:11:57.256378+0000","mapping_epoch":16,"log_start":"86'459518","ondisk_log_start":"86'459518","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":976,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 246s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6920175616,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":468828,"num_read_kb":1844496,"num_write":467217,"num_write_kb":1868868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3729,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:52.402745+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:11:52.402745+0000","last_peered":"2023-10-21T05:11:52.402745+0000","last_clean":"2023-10-21T05:11:52.402745+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:11:52.402745+0000","last_undegraded":"2023-10-21T05:11:52.402745+0000","last_fullsized":"2023-10-21T05:11:52.402745+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"88'439488","reported_seq":877450,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:11:55.874519+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:11:55.874519+0000","last_peered":"2023-10-21T05:11:55.874519+0000","last_clean":"2023-10-21T05:11:55.874519+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:11:55.874519+0000","last_undegraded":"2023-10-21T05:11:55.874519+0000","last_fullsized":"2023-10-21T05:11:55.874519+0000","mapping_epoch":16,"log_start":"86'429418","ondisk_log_start":"86'429418","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1409,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 260s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6412955648,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":439158,"num_read_kb":1727628,"num_write":437086,"num_write_kb":1748344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3664,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:52.396222+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:11:52.396222+0000","last_peered":"2023-10-21T05:11:52.396222+0000","last_clean":"2023-10-21T05:11:52.396222+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:11:52.396222+0000","last_undegraded":"2023-10-21T05:11:52.396222+0000","last_fullsized":"2023-10-21T05:11:52.396222+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"88'457683","reported_seq":912276,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:57.260482+0000","last_change":"2023-10-21T05:11:54.971614+0000","last_active":"2023-10-21T05:11:57.260482+0000","last_peered":"2023-10-21T05:11:57.260482+0000","last_clean":"2023-10-21T05:11:57.260482+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:11:57.260482+0000","last_undegraded":"2023-10-21T05:11:57.260482+0000","last_fullsized":"2023-10-21T05:11:57.260482+0000","mapping_epoch":16,"log_start":"86'447624","ondisk_log_start":"86'447624","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6731464723,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":456006,"num_read_kb":1793700,"num_write":455992,"num_write_kb":1823962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'15869","reported_seq":21940,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:57.105832+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:11:57.105832+0000","last_peered":"2023-10-21T05:11:57.105832+0000","last_clean":"2023-10-21T05:11:57.105832+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:11:57.105832+0000","last_undegraded":"2023-10-21T05:11:57.105832+0000","last_fullsized":"2023-10-21T05:11:57.105832+0000","mapping_epoch":13,"log_start":"80'5800","ondisk_log_start":"80'5800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":260,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:44.826644+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:11:44.826644+0000","last_peered":"2023-10-21T05:11:44.826644+0000","last_clean":"2023-10-21T05:11:44.826644+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:11:44.826644+0000","last_undegraded":"2023-10-21T05:11:44.826644+0000","last_fullsized":"2023-10-21T05:11:44.826644+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26870984723,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1826843,"num_read_kb":7187072,"num_write":1823432,"num_write_kb":7293722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14938497024,"data_stored":17832468518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40257,"ondisk_log_size":40257,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36563,"ondisk_log_size":36563,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739217,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9206504,"kb_used_data":7295008,"kb_used_omap":668,"kb_used_meta":1910819,"kb_avail":85165336,"statfs":{"total":96636764160,"available":87209304064,"internally_reserved":0,"allocated":7470088192,"data_stored":8916567213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":685008,"internal_metadata":1956678704},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.54600000000000004,"15min":0.44800000000000001},"min":{"1min":0.185,"5min":0.153,"15min":0.153},"max":{"1min":4.7699999999999996,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":2.6019999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.495,"15min":0.42599999999999999},"min":{"1min":0.23400000000000001,"5min":0.193,"15min":0.185},"max":{"1min":0.97199999999999998,"5min":4.319,"15min":4.319},"last":2.645}]},{"osd":1,"last update":"Sat Oct 21 05:11:46 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.66500000000000004,"15min":0.63500000000000001},"min":{"1min":0.21099999999999999,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":5.0709999999999997,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.52200000000000002,"15min":0.495},"min":{"1min":0.24299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.8170000000000002,"5min":4.8170000000000002,"15min":5.2199999999999998},"last":0.30399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739214,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5106276,"kb_used_data":3693596,"kb_used_omap":353,"kb_used_meta":1412318,"kb_avail":89265564,"statfs":{"total":96636764160,"available":91407937536,"internally_reserved":0,"allocated":3782242304,"data_stored":4528151807,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1446213847},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.58499999999999996,"15min":0.56000000000000005},"min":{"1min":0.19,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":3.633,"5min":4.5599999999999996,"15min":6.8289999999999997},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52900000000000003,"15min":0.55100000000000005},"min":{"1min":0.23999999999999999,"5min":0.186,"15min":0.17499999999999999},"max":{"1min":3.5870000000000002,"5min":4.6109999999999998,"15min":6.8460000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:11:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.216,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.60899999999999999,"5min":2.6440000000000001,"15min":4.2729999999999997},"last":8.7669999999999995},{"interface":"front","average":{"1min":0.378,"5min":0.39400000000000002,"15min":0.41099999999999998},"min":{"1min":0.221,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.70099999999999996,"5min":1.4590000000000001,"15min":5.3380000000000001},"last":0.51600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739217,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5448820,"kb_used_data":3601644,"kb_used_omap":329,"kb_used_meta":1846838,"kb_avail":88923020,"statfs":{"total":96636764160,"available":91057172480,"internally_reserved":0,"allocated":3688083456,"data_stored":4388980944,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1891162122},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,0,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:11:52 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.376,"15min":0.39400000000000002},"min":{"1min":0.185,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":1.3160000000000001,"5min":1.7430000000000001,"15min":4.0199999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.44,"5min":0.436,"15min":0.44800000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":1.3520000000000001,"5min":1.778,"15min":4.0919999999999996},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Oct 21 05:11:52 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.48299999999999998,"15min":0.45900000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.214},"max":{"1min":6.2830000000000004,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.55300000000000005,"15min":0.502},"min":{"1min":0.252,"5min":0.23499999999999999,"15min":0.22800000000000001},"max":{"1min":6.3710000000000004,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.51700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3687313408,"data_stored":4388249600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3781472256,"data_stored":4527521811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7469711360,"data_stored":8916697107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:11:58.336 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:58.336 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:58.337 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:58.337 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:11:58.337 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:12:06.237 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:06.231+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:12:06.238 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:06.231+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:12:06.238 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:06.231+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:12:06.239 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:06.231+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:12:06.253 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:06.247+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:12:06.257 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:06.255+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:12:06.260 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:06.255+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:12:06.260 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:06.255+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:12:18.338 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:12:18.972 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:12:18.973 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:12:19.028 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2211,"stamp":"2023-10-21T05:12:18.136320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26895805569,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1853115,"num_read_kb":7232049,"num_write":1831161,"num_write_kb":7323887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77053,"ondisk_log_size":77053,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19978612,"kb_used_data":14646300,"kb_used_omap":1352,"kb_used_meta":5330935,"kb_avail":263136908,"statfs":{"total":289910292480,"available":269452193792,"internally_reserved":0,"allocated":14997811200,"data_stored":17891097212,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1385199,"internal_metadata":5458877713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,10,9,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15364096,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4587,"num_read_kb":18316,"num_write":4430,"num_write_kb":17720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002106"},"pg_stats":[{"pgid":"2.7","version":"88'12762","reported_seq":17558,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:16.448797+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:12:16.448797+0000","last_peered":"2023-10-21T05:12:16.448797+0000","last_clean":"2023-10-21T05:12:16.448797+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:12:16.448797+0000","last_undegraded":"2023-10-21T05:12:16.448797+0000","last_fullsized":"2023-10-21T05:12:16.448797+0000","mapping_epoch":13,"log_start":"68'2700","ondisk_log_start":"68'2700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'6510","reported_seq":9945,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:16.440526+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:12:16.440526+0000","last_peered":"2023-10-21T05:12:16.440526+0000","last_clean":"2023-10-21T05:12:16.440526+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:12:16.440526+0000","last_undegraded":"2023-10-21T05:12:16.440526+0000","last_fullsized":"2023-10-21T05:12:16.440526+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6510,"log_dups_size":0,"ondisk_log_size":6510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"88'3261","reported_seq":4616,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:15.531516+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:12:15.531516+0000","last_peered":"2023-10-21T05:12:15.531516+0000","last_clean":"2023-10-21T05:12:15.531516+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:12:15.531516+0000","last_undegraded":"2023-10-21T05:12:15.531516+0000","last_fullsized":"2023-10-21T05:12:15.531516+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3261,"log_dups_size":0,"ondisk_log_size":3261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"88'6459","reported_seq":9304,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:16.437737+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:12:16.437737+0000","last_peered":"2023-10-21T05:12:16.437737+0000","last_clean":"2023-10-21T05:12:16.437737+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:12:16.437737+0000","last_undegraded":"2023-10-21T05:12:16.437737+0000","last_fullsized":"2023-10-21T05:12:16.437737+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6459,"log_dups_size":0,"ondisk_log_size":6459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"88'467181","reported_seq":930488,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:15.944309+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:12:15.944309+0000","last_peered":"2023-10-21T05:12:15.944309+0000","last_clean":"2023-10-21T05:12:15.944309+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:12:15.944309+0000","last_undegraded":"2023-10-21T05:12:15.944309+0000","last_fullsized":"2023-10-21T05:12:15.944309+0000","mapping_epoch":16,"log_start":"86'457116","ondisk_log_start":"86'457116","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1023,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 276s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6812372992,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":464635,"num_read_kb":1828360,"num_write":464881,"num_write_kb":1859524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":214,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:45.363521+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:10:45.363521+0000","last_peered":"2023-10-21T05:10:45.363521+0000","last_clean":"2023-10-21T05:10:45.363521+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:10:45.363521+0000","last_undegraded":"2023-10-21T05:10:45.363521+0000","last_fullsized":"2023-10-21T05:10:45.363521+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"88'471396","reported_seq":940596,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:17.316713+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:12:17.316713+0000","last_peered":"2023-10-21T05:12:17.316713+0000","last_clean":"2023-10-21T05:12:17.316713+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:12:17.316713+0000","last_undegraded":"2023-10-21T05:12:17.316713+0000","last_fullsized":"2023-10-21T05:12:17.316713+0000","mapping_epoch":16,"log_start":"86'461318","ondisk_log_start":"86'461318","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1064,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 266s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6926495744,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":470605,"num_read_kb":1851580,"num_write":468994,"num_write_kb":1875976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3741,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:12.878314+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:12:12.878314+0000","last_peered":"2023-10-21T05:12:12.878314+0000","last_clean":"2023-10-21T05:12:12.878314+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:12:12.878314+0000","last_undegraded":"2023-10-21T05:12:12.878314+0000","last_fullsized":"2023-10-21T05:12:12.878314+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"88'441189","reported_seq":880942,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:15.943956+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:12:15.943956+0000","last_peered":"2023-10-21T05:12:15.943956+0000","last_clean":"2023-10-21T05:12:15.943956+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:12:15.943956+0000","last_undegraded":"2023-10-21T05:12:15.943956+0000","last_fullsized":"2023-10-21T05:12:15.943956+0000","mapping_epoch":16,"log_start":"86'431118","ondisk_log_start":"86'431118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1525,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 280s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6418472960,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":440932,"num_read_kb":1734704,"num_write":438716,"num_write_kb":1754864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3677,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:12.876917+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:12:12.876917+0000","last_peered":"2023-10-21T05:12:12.876917+0000","last_clean":"2023-10-21T05:12:12.876917+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:12:12.876917+0000","last_undegraded":"2023-10-21T05:12:12.876917+0000","last_fullsized":"2023-10-21T05:12:12.876917+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"88'459523","reported_seq":915980,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:17.341120+0000","last_change":"2023-10-21T05:12:16.114314+0000","last_active":"2023-10-21T05:12:17.341120+0000","last_peered":"2023-10-21T05:12:17.341120+0000","last_clean":"2023-10-21T05:12:17.341120+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:12:17.341120+0000","last_undegraded":"2023-10-21T05:12:17.341120+0000","last_fullsized":"2023-10-21T05:12:17.341120+0000","mapping_epoch":16,"log_start":"86'449424","ondisk_log_start":"86'449424","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6737870867,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":457855,"num_read_kb":1801084,"num_write":457832,"num_write_kb":1831322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'15931","reported_seq":22022,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:16.490157+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:12:16.490157+0000","last_peered":"2023-10-21T05:12:16.490157+0000","last_clean":"2023-10-21T05:12:16.490157+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:12:16.490157+0000","last_undegraded":"2023-10-21T05:12:16.490157+0000","last_fullsized":"2023-10-21T05:12:16.490157+0000","mapping_epoch":13,"log_start":"80'5900","ondisk_log_start":"80'5900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":260,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:44.826644+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:11:44.826644+0000","last_peered":"2023-10-21T05:11:44.826644+0000","last_clean":"2023-10-21T05:11:44.826644+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:11:44.826644+0000","last_undegraded":"2023-10-21T05:11:44.826644+0000","last_fullsized":"2023-10-21T05:11:44.826644+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26895212563,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1834027,"num_read_kb":7215728,"num_write":1830423,"num_write_kb":7321686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14995808256,"data_stored":17889779750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40313,"ondisk_log_size":40313,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36641,"ondisk_log_size":36641,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739221,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9331608,"kb_used_data":7323024,"kb_used_omap":668,"kb_used_meta":2007907,"kb_avail":85040232,"statfs":{"total":96636764160,"available":87081197568,"internally_reserved":0,"allocated":7498776576,"data_stored":8945255597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":685008,"internal_metadata":2056096816},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.54600000000000004,"15min":0.44800000000000001},"min":{"1min":0.185,"5min":0.153,"15min":0.153},"max":{"1min":4.7699999999999996,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.495,"15min":0.42599999999999999},"min":{"1min":0.23400000000000001,"5min":0.193,"15min":0.185},"max":{"1min":0.97199999999999998,"5min":4.319,"15min":4.319},"last":0.36199999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:11:46 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.66500000000000004,"15min":0.63500000000000001},"min":{"1min":0.21099999999999999,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":5.0709999999999997,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.52200000000000002,"15min":0.495},"min":{"1min":0.24299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.8170000000000002,"5min":4.8170000000000002,"15min":5.2199999999999998},"last":0.48299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739218,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152768,"kb_used_data":3708152,"kb_used_omap":353,"kb_used_meta":1444254,"kb_avail":89219072,"statfs":{"total":96636764160,"available":91360329728,"internally_reserved":0,"allocated":3797147648,"data_stored":4543057151,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1478916311},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.58499999999999996,"15min":0.56000000000000005},"min":{"1min":0.19,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":3.633,"5min":4.5599999999999996,"15min":6.8289999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52900000000000003,"15min":0.55100000000000005},"min":{"1min":0.23999999999999999,"5min":0.186,"15min":0.17499999999999999},"max":{"1min":3.5870000000000002,"5min":4.6109999999999998,"15min":6.8460000000000001},"last":0.47599999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:11:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.216,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.60899999999999999,"5min":2.6440000000000001,"15min":4.2729999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.39400000000000002,"15min":0.41099999999999998},"min":{"1min":0.221,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.70099999999999996,"5min":1.4590000000000001,"15min":5.3380000000000001},"last":0.51300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739221,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5494236,"kb_used_data":3615124,"kb_used_omap":329,"kb_used_meta":1878774,"kb_avail":88877604,"statfs":{"total":96636764160,"available":91010666496,"internally_reserved":0,"allocated":3701886976,"data_stored":4402784464,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1923864586},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:11:52 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.376,"15min":0.39400000000000002},"min":{"1min":0.185,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":1.3160000000000001,"5min":1.7430000000000001,"15min":4.0199999999999996},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.436,"15min":0.44800000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":1.3520000000000001,"5min":1.778,"15min":4.0919999999999996},"last":0.45900000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:11:52 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.48299999999999998,"15min":0.45900000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.214},"max":{"1min":6.2830000000000004,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.55300000000000005,"15min":0.502},"min":{"1min":0.252,"5min":0.23499999999999999,"15min":0.22800000000000001},"max":{"1min":6.3710000000000004,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.42799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3701133312,"data_stored":4402069504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3796283392,"data_stored":4542332947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7498391552,"data_stored":8945377299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:12:19.030 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:19.030 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:19.031 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:19.031 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:19.031 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:12:19.697 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-21T05:12:19.746 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:12:20.316 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:12:20.353 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:12:20.876 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-21T05:12:20.905 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:12:21.464 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:12:21.497 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:12:36.119 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.115+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:12:36.120 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.115+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:12:36.149 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.143+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:12:36.149 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.143+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:12:36.179 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.175+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:12:36.180 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.175+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:12:36.180 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.175+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:12:36.186 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.179+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:12:36.186 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.179+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:12:36.201 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.195+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:12:36.205 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.199+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:12:36.206 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.199+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:12:36.206 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:36.199+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:12:40.763 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:40.759+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:12:41.501 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:12:42.078 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:12:42.079 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:12:42.108 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2222,"stamp":"2023-10-21T05:12:40.140358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26920242305,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1860217,"num_read_kb":7260365,"num_write":1838183,"num_write_kb":7351975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76948,"ondisk_log_size":76948,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20187032,"kb_used_data":14702012,"kb_used_omap":1352,"kb_used_meta":5483639,"kb_avail":262928488,"statfs":{"total":289910292480,"available":269238771712,"internally_reserved":0,"allocated":15054860288,"data_stored":17948146300,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1385199,"internal_metadata":5615246609},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,6,9,19,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12288000,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3595,"num_read_kb":14332,"num_write":3536,"num_write_kb":14144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002315"},"pg_stats":[{"pgid":"2.7","version":"88'12824","reported_seq":17635,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:37.235738+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:12:37.235738+0000","last_peered":"2023-10-21T05:12:37.235738+0000","last_clean":"2023-10-21T05:12:37.235738+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:12:37.235738+0000","last_undegraded":"2023-10-21T05:12:37.235738+0000","last_fullsized":"2023-10-21T05:12:37.235738+0000","mapping_epoch":13,"log_start":"70'2800","ondisk_log_start":"70'2800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'6542","reported_seq":9985,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:37.214627+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:12:37.214627+0000","last_peered":"2023-10-21T05:12:37.214627+0000","last_clean":"2023-10-21T05:12:37.214627+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:12:37.214627+0000","last_undegraded":"2023-10-21T05:12:37.214627+0000","last_fullsized":"2023-10-21T05:12:37.214627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6542,"log_dups_size":0,"ondisk_log_size":6542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"88'3277","reported_seq":4636,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:36.168443+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:12:36.168443+0000","last_peered":"2023-10-21T05:12:36.168443+0000","last_clean":"2023-10-21T05:12:36.168443+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:12:36.168443+0000","last_undegraded":"2023-10-21T05:12:36.168443+0000","last_fullsized":"2023-10-21T05:12:36.168443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"88'6490","reported_seq":9342,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:37.223812+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:12:37.223812+0000","last_peered":"2023-10-21T05:12:37.223812+0000","last_clean":"2023-10-21T05:12:37.223812+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:12:37.223812+0000","last_undegraded":"2023-10-21T05:12:37.223812+0000","last_fullsized":"2023-10-21T05:12:37.223812+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"88'469032","reported_seq":934167,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:36.087757+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:12:36.087757+0000","last_peered":"2023-10-21T05:12:36.087757+0000","last_clean":"2023-10-21T05:12:36.087757+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:12:36.087757+0000","last_undegraded":"2023-10-21T05:12:36.087757+0000","last_fullsized":"2023-10-21T05:12:36.087757+0000","mapping_epoch":16,"log_start":"87'459017","ondisk_log_start":"87'459017","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1111,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 296s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6818537472,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":466458,"num_read_kb":1835636,"num_write":466676,"num_write_kb":1866704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":214,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-21T05:10:45.363521+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:10:45.363521+0000","last_peered":"2023-10-21T05:10:45.363521+0000","last_clean":"2023-10-21T05:10:45.363521+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:10:45.363521+0000","last_undegraded":"2023-10-21T05:10:45.363521+0000","last_fullsized":"2023-10-21T05:10:45.363521+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"88'473205","reported_seq":944244,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:37.447464+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:12:37.447464+0000","last_peered":"2023-10-21T05:12:37.447464+0000","last_clean":"2023-10-21T05:12:37.447464+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:12:37.447464+0000","last_undegraded":"2023-10-21T05:12:37.447464+0000","last_fullsized":"2023-10-21T05:12:37.447464+0000","mapping_epoch":16,"log_start":"87'463118","ondisk_log_start":"87'463118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1144,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 286s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6932717568,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":472433,"num_read_kb":1858864,"num_write":470754,"num_write_kb":1883016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3754,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:33.382301+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:12:33.382301+0000","last_peered":"2023-10-21T05:12:33.382301+0000","last_clean":"2023-10-21T05:12:33.382301+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:12:33.382301+0000","last_undegraded":"2023-10-21T05:12:33.382301+0000","last_fullsized":"2023-10-21T05:12:33.382301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"88'442931","reported_seq":884380,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:36.044399+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:12:36.044399+0000","last_peered":"2023-10-21T05:12:36.044399+0000","last_clean":"2023-10-21T05:12:36.044399+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:12:36.044399+0000","last_undegraded":"2023-10-21T05:12:36.044399+0000","last_fullsized":"2023-10-21T05:12:36.044399+0000","mapping_epoch":16,"log_start":"87'432918","ondisk_log_start":"87'432918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1614,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 300s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6424199168,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":442611,"num_read_kb":1741396,"num_write":440405,"num_write_kb":1761620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3689,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:33.358494+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:12:33.358494+0000","last_peered":"2023-10-21T05:12:33.358494+0000","last_clean":"2023-10-21T05:12:33.358494+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:12:33.358494+0000","last_undegraded":"2023-10-21T05:12:33.358494+0000","last_fullsized":"2023-10-21T05:12:33.358494+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"88'461301","reported_seq":919539,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:37.462504+0000","last_change":"2023-10-21T05:12:37.289210+0000","last_active":"2023-10-21T05:12:37.462504+0000","last_peered":"2023-10-21T05:12:37.462504+0000","last_clean":"2023-10-21T05:12:37.462504+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:12:37.462504+0000","last_undegraded":"2023-10-21T05:12:37.462504+0000","last_fullsized":"2023-10-21T05:12:37.462504+0000","mapping_epoch":16,"log_start":"87'451224","ondisk_log_start":"87'451224","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6744195091,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":459627,"num_read_kb":1808148,"num_write":459610,"num_write_kb":1838434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'16006","reported_seq":22114,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:12:37.307383+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:12:37.307383+0000","last_peered":"2023-10-21T05:12:37.307383+0000","last_clean":"2023-10-21T05:12:37.307383+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:12:37.307383+0000","last_undegraded":"2023-10-21T05:12:37.307383+0000","last_fullsized":"2023-10-21T05:12:37.307383+0000","mapping_epoch":13,"log_start":"80'6000","ondisk_log_start":"80'6000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":260,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:44.826644+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:11:44.826644+0000","last_peered":"2023-10-21T05:11:44.826644+0000","last_clean":"2023-10-21T05:11:44.826644+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:11:44.826644+0000","last_undegraded":"2023-10-21T05:11:44.826644+0000","last_fullsized":"2023-10-21T05:11:44.826644+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26919649299,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1841129,"num_read_kb":7244044,"num_write":1837445,"num_write_kb":7349774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15053303808,"data_stored":17947275302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40192,"ondisk_log_size":40192,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36657,"ondisk_log_size":36657,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739225,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9439920,"kb_used_data":7350820,"kb_used_omap":668,"kb_used_meta":2088419,"kb_avail":84931920,"statfs":{"total":96636764160,"available":86970286080,"internally_reserved":0,"allocated":7527239680,"data_stored":8973718701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":685008,"internal_metadata":2138541104},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,5,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.54600000000000004,"15min":0.44800000000000001},"min":{"1min":0.185,"5min":0.153,"15min":0.153},"max":{"1min":4.7699999999999996,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.495,"15min":0.42599999999999999},"min":{"1min":0.23400000000000001,"5min":0.193,"15min":0.185},"max":{"1min":0.97199999999999998,"5min":4.319,"15min":4.319},"last":0.29899999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:11:46 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.66500000000000004,"15min":0.63500000000000001},"min":{"1min":0.21099999999999999,"5min":0.20499999999999999,"15min":0.192},"max":{"1min":5.0709999999999997,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.52200000000000002,"15min":0.495},"min":{"1min":0.24299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.8170000000000002,"5min":4.8170000000000002,"15min":5.2199999999999998},"last":0.47499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739222,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5203040,"kb_used_data":3722328,"kb_used_omap":353,"kb_used_meta":1480350,"kb_avail":89168800,"statfs":{"total":96636764160,"available":91308851200,"internally_reserved":0,"allocated":3811663872,"data_stored":4557573375,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1515878615},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:11:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.58499999999999996,"15min":0.56000000000000005},"min":{"1min":0.19,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":3.633,"5min":4.5599999999999996,"15min":6.8289999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52900000000000003,"15min":0.55100000000000005},"min":{"1min":0.23999999999999999,"5min":0.186,"15min":0.17499999999999999},"max":{"1min":3.5870000000000002,"5min":4.6109999999999998,"15min":6.8460000000000001},"last":0.49099999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:11:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.216,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":0.60899999999999999,"5min":2.6440000000000001,"15min":4.2729999999999997},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.39400000000000002,"15min":0.41099999999999998},"min":{"1min":0.221,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.70099999999999996,"5min":1.4590000000000001,"15min":5.3380000000000001},"last":0.32300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739225,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5544072,"kb_used_data":3628864,"kb_used_omap":329,"kb_used_meta":1914870,"kb_avail":88827768,"statfs":{"total":96636764160,"available":90959634432,"internally_reserved":0,"allocated":3715956736,"data_stored":4416854224,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1960826890},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,8,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:11:52 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.376,"15min":0.39400000000000002},"min":{"1min":0.185,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":1.3160000000000001,"5min":1.7430000000000001,"15min":4.0199999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.436,"15min":0.44800000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":1.3520000000000001,"5min":1.778,"15min":4.0919999999999996},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:11:52 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.48299999999999998,"15min":0.45900000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.214},"max":{"1min":6.2830000000000004,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":2.411},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.55300000000000005,"15min":0.502},"min":{"1min":0.252,"5min":0.23499999999999999,"15min":0.22800000000000001},"max":{"1min":6.3710000000000004,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":2.4849999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3715411968,"data_stored":4416348160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3810770944,"data_stored":4556820499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7527120896,"data_stored":8974106643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:12:42.109 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:42.110 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:42.110 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:42.110 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:12:42.111 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:12:42.519 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T05:12:42.520 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:12:42.520 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:12:42.520 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:12:42.521 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:12:42.521 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T05:12:42.521 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:12:42.522 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:12:42.522 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:12:42.522 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:12:42.522 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:12:42.523 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:12:42.523 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:12:42.525 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:42.515+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:12:57.136 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:12:57.131+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:13:02.112 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:13:02.626 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:13:02.626 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:13:02.662 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2234,"stamp":"2023-10-21T05:13:02.144187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26947357825,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1868150,"num_read_kb":7291949,"num_write":1845997,"num_write_kb":7383231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77244,"ondisk_log_size":77244,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19188984,"kb_used_data":14761304,"kb_used_omap":1470,"kb_used_meta":4426177,"kb_avail":263926536,"statfs":{"total":289910292480,"available":270260772864,"internally_reserved":0,"allocated":15115575296,"data_stored":18008848234,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1505316,"internal_metadata":4532406236},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,7,11,12,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15745024,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4575,"num_read_kb":18204,"num_write":4511,"num_write_kb":18044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002065"},"pg_stats":[{"pgid":"2.7","version":"89'12881","reported_seq":17709,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:57.175445+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:12:57.175445+0000","last_peered":"2023-10-21T05:12:57.175445+0000","last_clean":"2023-10-21T05:12:57.175445+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:12:57.175445+0000","last_undegraded":"2023-10-21T05:12:57.175445+0000","last_fullsized":"2023-10-21T05:12:57.175445+0000","mapping_epoch":13,"log_start":"70'2800","ondisk_log_start":"70'2800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"89'6573","reported_seq":10025,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:57.153964+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:12:57.153964+0000","last_peered":"2023-10-21T05:12:57.153964+0000","last_clean":"2023-10-21T05:12:57.153964+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:12:57.153964+0000","last_undegraded":"2023-10-21T05:12:57.153964+0000","last_fullsized":"2023-10-21T05:12:57.153964+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6573,"log_dups_size":0,"ondisk_log_size":6573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'3293","reported_seq":4658,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:57.130681+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:12:57.130681+0000","last_peered":"2023-10-21T05:12:57.130681+0000","last_clean":"2023-10-21T05:12:57.130681+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:12:57.130681+0000","last_undegraded":"2023-10-21T05:12:57.130681+0000","last_fullsized":"2023-10-21T05:12:57.130681+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3293,"log_dups_size":0,"ondisk_log_size":3293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'6520","reported_seq":9382,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:57.153921+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:12:57.153921+0000","last_peered":"2023-10-21T05:12:57.153921+0000","last_clean":"2023-10-21T05:12:57.153921+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:12:57.153921+0000","last_undegraded":"2023-10-21T05:12:57.153921+0000","last_fullsized":"2023-10-21T05:12:57.153921+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6520,"log_dups_size":0,"ondisk_log_size":6520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'471287","reported_seq":938704,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:01.255205+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:13:01.255205+0000","last_peered":"2023-10-21T05:13:01.255205+0000","last_clean":"2023-10-21T05:13:01.255205+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:13:01.255205+0000","last_undegraded":"2023-10-21T05:13:01.255205+0000","last_fullsized":"2023-10-21T05:13:01.255205+0000","mapping_epoch":16,"log_start":"87'461217","ondisk_log_start":"87'461217","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1248,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 321s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6826041344,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":468722,"num_read_kb":1844640,"num_write":468847,"num_write_kb":1875388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":216,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:44.846014+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:12:44.846014+0000","last_peered":"2023-10-21T05:12:44.846014+0000","last_clean":"2023-10-21T05:12:44.846014+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:12:44.846014+0000","last_undegraded":"2023-10-21T05:12:44.846014+0000","last_fullsized":"2023-10-21T05:12:44.846014+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'475092","reported_seq":948012,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:57.615349+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:12:57.615349+0000","last_peered":"2023-10-21T05:12:57.615349+0000","last_clean":"2023-10-21T05:12:57.615349+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:12:57.615349+0000","last_undegraded":"2023-10-21T05:12:57.615349+0000","last_fullsized":"2023-10-21T05:12:57.615349+0000","mapping_epoch":16,"log_start":"87'465018","ondisk_log_start":"87'465018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1188,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 306s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6939357184,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":474303,"num_read_kb":1866320,"num_write":472614,"num_write_kb":1890456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3767,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:53.861746+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:12:53.861746+0000","last_peered":"2023-10-21T05:12:53.861746+0000","last_clean":"2023-10-21T05:12:53.861746+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:12:53.861746+0000","last_undegraded":"2023-10-21T05:12:53.861746+0000","last_fullsized":"2023-10-21T05:12:53.861746+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'445080","reported_seq":888644,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:01.255823+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:13:01.255823+0000","last_peered":"2023-10-21T05:13:01.255823+0000","last_clean":"2023-10-21T05:13:01.255823+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:13:01.255823+0000","last_undegraded":"2023-10-21T05:13:01.255823+0000","last_fullsized":"2023-10-21T05:13:01.255823+0000","mapping_epoch":16,"log_start":"87'435019","ondisk_log_start":"87'435019","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1731,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 325s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6431252480,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":444709,"num_read_kb":1749748,"num_write":442488,"num_write_kb":1769952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3702,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:53.832809+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:12:53.832809+0000","last_peered":"2023-10-21T05:12:53.832809+0000","last_clean":"2023-10-21T05:12:53.832809+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:12:53.832809+0000","last_undegraded":"2023-10-21T05:12:53.832809+0000","last_fullsized":"2023-10-21T05:12:53.832809+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'463001","reported_seq":922947,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:12:57.631799+0000","last_change":"2023-10-21T05:12:57.604455+0000","last_active":"2023-10-21T05:12:57.631799+0000","last_peered":"2023-10-21T05:12:57.631799+0000","last_clean":"2023-10-21T05:12:57.631799+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:12:57.631799+0000","last_undegraded":"2023-10-21T05:12:57.631799+0000","last_fullsized":"2023-10-21T05:12:57.631799+0000","mapping_epoch":16,"log_start":"87'452924","ondisk_log_start":"87'452924","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6750113811,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":461328,"num_read_kb":1814920,"num_write":461310,"num_write_kb":1845234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'16078","reported_seq":22208,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:57.186801+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:12:57.186801+0000","last_peered":"2023-10-21T05:12:57.186801+0000","last_clean":"2023-10-21T05:12:57.186801+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:12:57.186801+0000","last_undegraded":"2023-10-21T05:12:57.186801+0000","last_fullsized":"2023-10-21T05:12:57.186801+0000","mapping_epoch":13,"log_start":"80'6000","ondisk_log_start":"80'6000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":261,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:57.264170+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:11:57.264170+0000","last_peered":"2023-10-21T05:11:57.264170+0000","last_clean":"2023-10-21T05:11:57.264170+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:11:57.264170+0000","last_undegraded":"2023-10-21T05:11:57.264170+0000","last_fullsized":"2023-10-21T05:11:57.264170+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26946764819,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1849062,"num_read_kb":7275628,"num_write":1845259,"num_write_kb":7381030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15113760768,"data_stored":18007732262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40282,"ondisk_log_size":40282,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36863,"ondisk_log_size":36863,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739229,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8320208,"kb_used_data":7378696,"kb_used_omap":786,"kb_used_meta":940717,"kb_avail":86051632,"statfs":{"total":96636764160,"available":88116871168,"internally_reserved":0,"allocated":7555784704,"data_stored":9002259367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":963294971},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.45100000000000001},"min":{"1min":0.17899999999999999,"5min":0.153,"15min":0.153},"max":{"1min":2.6019999999999999,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":4.8540000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.54700000000000004,"15min":0.43099999999999999},"min":{"1min":0.22700000000000001,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":2.645,"5min":4.319,"15min":4.319},"last":0.215}]},{"osd":1,"last update":"Sat Oct 21 05:12:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.62,"15min":0.62},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.192},"max":{"1min":0.71199999999999997,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.53000000000000003,"15min":0.48699999999999999},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":4.8170000000000002,"15min":5.2199999999999998},"last":4.8979999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739226,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5253500,"kb_used_data":3736688,"kb_used_omap":353,"kb_used_meta":1516446,"kb_avail":89118340,"statfs":{"total":96636764160,"available":91257180160,"internally_reserved":0,"allocated":3826368512,"data_stored":4572273657,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1552840919},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,0,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.57199999999999995,"15min":0.54700000000000004},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":0.67000000000000004,"5min":4.5599999999999996,"15min":6.8289999999999997},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.25900000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.69999999999999996,"5min":4.6109999999999998,"15min":6.8460000000000001},"last":0.27400000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:12:44 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.47399999999999998,"15min":0.45800000000000002},"min":{"1min":0.22600000000000001,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":8.7669999999999995,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38700000000000001,"15min":0.39600000000000002},"min":{"1min":0.26600000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.748,"5min":1.4590000000000001,"15min":4.306},"last":0.314}]}]},{"osd":0,"up_from":8,"seq":34359739230,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5615276,"kb_used_data":3645920,"kb_used_omap":329,"kb_used_meta":1969014,"kb_avail":88756564,"statfs":{"total":96636764160,"available":90886721536,"internally_reserved":0,"allocated":3733422080,"data_stored":4434315210,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":2016270346},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,6,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.371,"15min":0.39100000000000001},"min":{"1min":0.247,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":0.56799999999999995,"5min":1.7430000000000001,"15min":4.0199999999999996},"last":0.376},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.42299999999999999,"15min":0.44400000000000001},"min":{"1min":0.28599999999999998,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":0.60399999999999998,"5min":1.778,"15min":4.0919999999999996},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.49399999999999999,"15min":0.45700000000000002},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.214},"max":{"1min":2.411,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57099999999999995,"15min":0.504},"min":{"1min":0.251,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.4849999999999999,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.76600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3732828160,"data_stored":4433764352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3825373184,"data_stored":4571422739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7555559424,"data_stored":9002545171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:13:02.664 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:02.664 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:02.664 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:02.665 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:02.665 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:13:22.667 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:13:23.151 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:13:23.152 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:13:23.185 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2244,"stamp":"2023-10-21T05:13:22.147897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26972114049,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1875426,"num_read_kb":7320893,"num_write":1853104,"num_write_kb":7411659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77242,"ondisk_log_size":77242,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19398608,"kb_used_data":14818232,"kb_used_omap":1470,"kb_used_meta":4578881,"kb_avail":263716912,"statfs":{"total":289910292480,"available":270046117888,"internally_reserved":0,"allocated":15173869568,"data_stored":18067142506,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1505316,"internal_metadata":4688775132},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,8,17,18,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15450112,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4456,"num_read_kb":17720,"num_write":4424,"num_write_kb":17696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002138"},"pg_stats":[{"pgid":"2.7","version":"89'12949","reported_seq":17793,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:16.497259+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:13:16.497259+0000","last_peered":"2023-10-21T05:13:16.497259+0000","last_clean":"2023-10-21T05:13:16.497259+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:13:16.497259+0000","last_undegraded":"2023-10-21T05:13:16.497259+0000","last_fullsized":"2023-10-21T05:13:16.497259+0000","mapping_epoch":13,"log_start":"70'2900","ondisk_log_start":"70'2900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"89'6607","reported_seq":10067,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:16.485034+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:13:16.485034+0000","last_peered":"2023-10-21T05:13:16.485034+0000","last_clean":"2023-10-21T05:13:16.485034+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:13:16.485034+0000","last_undegraded":"2023-10-21T05:13:16.485034+0000","last_fullsized":"2023-10-21T05:13:16.485034+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6607,"log_dups_size":0,"ondisk_log_size":6607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'3310","reported_seq":4679,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:16.220247+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:13:16.220247+0000","last_peered":"2023-10-21T05:13:16.220247+0000","last_clean":"2023-10-21T05:13:16.220247+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:13:16.220247+0000","last_undegraded":"2023-10-21T05:13:16.220247+0000","last_fullsized":"2023-10-21T05:13:16.220247+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3310,"log_dups_size":0,"ondisk_log_size":3310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'6554","reported_seq":9424,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:16.476409+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:13:16.476409+0000","last_peered":"2023-10-21T05:13:16.476409+0000","last_clean":"2023-10-21T05:13:16.476409+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:13:16.476409+0000","last_undegraded":"2023-10-21T05:13:16.476409+0000","last_fullsized":"2023-10-21T05:13:16.476409+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6554,"log_dups_size":0,"ondisk_log_size":6554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'473091","reported_seq":942345,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:21.378037+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:13:21.378037+0000","last_peered":"2023-10-21T05:13:21.378037+0000","last_clean":"2023-10-21T05:13:21.378037+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:13:21.378037+0000","last_undegraded":"2023-10-21T05:13:21.378037+0000","last_fullsized":"2023-10-21T05:13:21.378037+0000","mapping_epoch":16,"log_start":"87'463017","ondisk_log_start":"87'463017","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1344,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 341s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6832066560,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":470547,"num_read_kb":1851916,"num_write":470593,"num_write_kb":1882372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":216,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:44.846014+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:12:44.846014+0000","last_peered":"2023-10-21T05:12:44.846014+0000","last_clean":"2023-10-21T05:12:44.846014+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:12:44.846014+0000","last_undegraded":"2023-10-21T05:12:44.846014+0000","last_fullsized":"2023-10-21T05:12:44.846014+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'476974","reported_seq":951774,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:17.779110+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:13:17.779110+0000","last_peered":"2023-10-21T05:13:17.779110+0000","last_clean":"2023-10-21T05:13:17.779110+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:13:17.779110+0000","last_undegraded":"2023-10-21T05:13:17.779110+0000","last_fullsized":"2023-10-21T05:13:17.779110+0000","mapping_epoch":16,"log_start":"87'466918","ondisk_log_start":"87'466918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1254,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 327s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6945914880,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":476174,"num_read_kb":1873760,"num_write":474455,"num_write_kb":1897820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3779,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:14.312435+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:13:14.312435+0000","last_peered":"2023-10-21T05:13:14.312435+0000","last_clean":"2023-10-21T05:13:14.312435+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:13:14.312435+0000","last_undegraded":"2023-10-21T05:13:14.312435+0000","last_fullsized":"2023-10-21T05:13:14.312435+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'446831","reported_seq":892166,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:21.346920+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:13:21.346920+0000","last_peered":"2023-10-21T05:13:21.346920+0000","last_clean":"2023-10-21T05:13:21.346920+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:13:21.346920+0000","last_undegraded":"2023-10-21T05:13:21.346920+0000","last_fullsized":"2023-10-21T05:13:21.346920+0000","mapping_epoch":16,"log_start":"87'436819","ondisk_log_start":"87'436819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1818,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 345s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6437044224,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":446469,"num_read_kb":1756752,"num_write":444185,"num_write_kb":1776740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3714,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:14.313173+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:13:14.313173+0000","last_peered":"2023-10-21T05:13:14.313173+0000","last_clean":"2023-10-21T05:13:14.313173+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:13:14.313173+0000","last_undegraded":"2023-10-21T05:13:14.313173+0000","last_fullsized":"2023-10-21T05:13:14.313173+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'464824","reported_seq":926595,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:17.781244+0000","last_change":"2023-10-21T05:13:17.786445+0000","last_active":"2023-10-21T05:13:17.781244+0000","last_peered":"2023-10-21T05:13:17.781244+0000","last_clean":"2023-10-21T05:13:17.781244+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:13:17.781244+0000","last_undegraded":"2023-10-21T05:13:17.781244+0000","last_fullsized":"2023-10-21T05:13:17.781244+0000","mapping_epoch":16,"log_start":"87'454724","ondisk_log_start":"87'454724","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6756495379,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":463148,"num_read_kb":1822144,"num_write":463133,"num_write_kb":1852526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'16163","reported_seq":22313,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:16.542359+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:13:16.542359+0000","last_peered":"2023-10-21T05:13:16.542359+0000","last_clean":"2023-10-21T05:13:16.542359+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:13:16.542359+0000","last_undegraded":"2023-10-21T05:13:16.542359+0000","last_fullsized":"2023-10-21T05:13:16.542359+0000","mapping_epoch":13,"log_start":"80'6100","ondisk_log_start":"80'6100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":261,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:57.264170+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:11:57.264170+0000","last_peered":"2023-10-21T05:11:57.264170+0000","last_clean":"2023-10-21T05:11:57.264170+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:11:57.264170+0000","last_undegraded":"2023-10-21T05:11:57.264170+0000","last_fullsized":"2023-10-21T05:11:57.264170+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26971521043,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1856338,"num_read_kb":7304572,"num_write":1852366,"num_write_kb":7409458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15171915776,"data_stored":18065887270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40242,"ondisk_log_size":40242,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36901,"ondisk_log_size":36901,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739233,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8451344,"kb_used_data":7407112,"kb_used_omap":786,"kb_used_meta":1043437,"kb_avail":85920496,"statfs":{"total":96636764160,"available":87982587904,"internally_reserved":0,"allocated":7584882688,"data_stored":9031357351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":1068480251},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.45100000000000001},"min":{"1min":0.17899999999999999,"5min":0.153,"15min":0.153},"max":{"1min":2.6019999999999999,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.54700000000000004,"15min":0.43099999999999999},"min":{"1min":0.22700000000000001,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":2.645,"5min":4.319,"15min":4.319},"last":0.28999999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:12:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.62,"15min":0.62},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.192},"max":{"1min":0.71199999999999997,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.53000000000000003,"15min":0.48699999999999999},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":4.8170000000000002,"15min":5.2199999999999998},"last":0.60399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739230,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5300116,"kb_used_data":3751372,"kb_used_omap":353,"kb_used_meta":1548382,"kb_avail":89071724,"statfs":{"total":96636764160,"available":91209445376,"internally_reserved":0,"allocated":3841404928,"data_stored":4587310073,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1585543383},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.57199999999999995,"15min":0.54700000000000004},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":0.67000000000000004,"5min":4.5599999999999996,"15min":6.8289999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.25900000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.69999999999999996,"5min":4.6109999999999998,"15min":6.8460000000000001},"last":0.20200000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:12:44 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.47399999999999998,"15min":0.45800000000000002},"min":{"1min":0.22600000000000001,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":8.7669999999999995,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38700000000000001,"15min":0.39600000000000002},"min":{"1min":0.26600000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.748,"5min":1.4590000000000001,"15min":4.306},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739234,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5647148,"kb_used_data":3659748,"kb_used_omap":329,"kb_used_meta":1987062,"kb_avail":88724692,"statfs":{"total":96636764160,"available":90854084608,"internally_reserved":0,"allocated":3747581952,"data_stored":4448475082,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":2034751498},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,11,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.371,"15min":0.39100000000000001},"min":{"1min":0.247,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":0.56799999999999995,"5min":1.7430000000000001,"15min":4.0199999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.42299999999999999,"15min":0.44400000000000001},"min":{"1min":0.28599999999999998,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":0.60399999999999998,"5min":1.778,"15min":4.0919999999999996},"last":0.46899999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.49399999999999999,"15min":0.45700000000000002},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.214},"max":{"1min":2.411,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":0.307},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57099999999999995,"15min":0.504},"min":{"1min":0.251,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.4849999999999999,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.504}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3746942976,"data_stored":4447879168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3840376832,"data_stored":4586426387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7584595968,"data_stored":9031581715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:13:23.187 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:23.187 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:23.187 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:23.188 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:23.188 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:13:23.380 INFO:tasks.ceph.mgr.x.smithi133.stderr:2023-10-21T05:13:23.374+0000 7fd1fcf37640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-21T05:13:43.189 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:13:43.791 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:13:43.792 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:13:43.845 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2254,"stamp":"2023-10-21T05:13:42.153532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26996124801,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1882593,"num_read_kb":7349405,"num_write":1860042,"num_write_kb":7439411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77286,"ondisk_log_size":77286,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19127068,"kb_used_data":14872644,"kb_used_omap":1470,"kb_used_meta":4252929,"kb_avail":263988452,"statfs":{"total":289910292480,"available":270324174848,"internally_reserved":0,"allocated":15229587456,"data_stored":18122860394,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1505316,"internal_metadata":4355000284},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,3,1,8,18,24,20],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14663680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4369,"num_read_kb":17376,"num_write":4250,"num_write_kb":17000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004192"},"pg_stats":[{"pgid":"2.7","version":"89'12981","reported_seq":17842,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:37.868642+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:13:37.868642+0000","last_peered":"2023-10-21T05:13:37.868642+0000","last_clean":"2023-10-21T05:13:37.868642+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:13:37.868642+0000","last_undegraded":"2023-10-21T05:13:37.868642+0000","last_fullsized":"2023-10-21T05:13:37.868642+0000","mapping_epoch":13,"log_start":"70'2900","ondisk_log_start":"70'2900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"89'6623","reported_seq":10092,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:37.822111+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:13:37.822111+0000","last_peered":"2023-10-21T05:13:37.822111+0000","last_clean":"2023-10-21T05:13:37.822111+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:13:37.822111+0000","last_undegraded":"2023-10-21T05:13:37.822111+0000","last_fullsized":"2023-10-21T05:13:37.822111+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6623,"log_dups_size":0,"ondisk_log_size":6623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'3318","reported_seq":4691,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:35.711825+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:13:35.711825+0000","last_peered":"2023-10-21T05:13:35.711825+0000","last_clean":"2023-10-21T05:13:35.711825+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:13:35.711825+0000","last_undegraded":"2023-10-21T05:13:35.711825+0000","last_fullsized":"2023-10-21T05:13:35.711825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3318,"log_dups_size":0,"ondisk_log_size":3318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'6570","reported_seq":9448,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:37.828157+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:13:37.828157+0000","last_peered":"2023-10-21T05:13:37.828157+0000","last_clean":"2023-10-21T05:13:37.828157+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:13:37.828157+0000","last_undegraded":"2023-10-21T05:13:37.828157+0000","last_fullsized":"2023-10-21T05:13:37.828157+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6570,"log_dups_size":0,"ondisk_log_size":6570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'474912","reported_seq":946013,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:41.480784+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:13:41.480784+0000","last_peered":"2023-10-21T05:13:41.480784+0000","last_clean":"2023-10-21T05:13:41.480784+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:13:41.480784+0000","last_undegraded":"2023-10-21T05:13:41.480784+0000","last_fullsized":"2023-10-21T05:13:41.480784+0000","mapping_epoch":16,"log_start":"88'464817","ondisk_log_start":"88'464817","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1424,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 361s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6838272000,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":472383,"num_read_kb":1859220,"num_write":472368,"num_write_kb":1889472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":216,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:44.846014+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:12:44.846014+0000","last_peered":"2023-10-21T05:12:44.846014+0000","last_clean":"2023-10-21T05:12:44.846014+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:12:44.846014+0000","last_undegraded":"2023-10-21T05:12:44.846014+0000","last_fullsized":"2023-10-21T05:12:44.846014+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'478804","reported_seq":955435,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:37.931586+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:13:37.931586+0000","last_peered":"2023-10-21T05:13:37.931586+0000","last_clean":"2023-10-21T05:13:37.931586+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:13:37.931586+0000","last_undegraded":"2023-10-21T05:13:37.931586+0000","last_fullsized":"2023-10-21T05:13:37.931586+0000","mapping_epoch":16,"log_start":"88'468718","ondisk_log_start":"88'468718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1372,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 347s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6952038400,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":478001,"num_read_kb":1881028,"num_write":476213,"num_write_kb":1904852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3791,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:34.792534+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:13:34.792534+0000","last_peered":"2023-10-21T05:13:34.792534+0000","last_clean":"2023-10-21T05:13:34.792534+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:13:34.792534+0000","last_undegraded":"2023-10-21T05:13:34.792534+0000","last_fullsized":"2023-10-21T05:13:34.792534+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'448598","reported_seq":895645,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:13:41.480460+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:13:41.480460+0000","last_peered":"2023-10-21T05:13:41.480460+0000","last_clean":"2023-10-21T05:13:41.480460+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:13:41.480460+0000","last_undegraded":"2023-10-21T05:13:41.480460+0000","last_fullsized":"2023-10-21T05:13:41.480460+0000","mapping_epoch":16,"log_start":"88'438519","ondisk_log_start":"88'438519","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1940,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 365s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6442704896,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":448175,"num_read_kb":1763552,"num_write":445876,"num_write_kb":1783504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3726,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:34.802849+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:13:34.802849+0000","last_peered":"2023-10-21T05:13:34.802849+0000","last_clean":"2023-10-21T05:13:34.802849+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:13:34.802849+0000","last_undegraded":"2023-10-21T05:13:34.802849+0000","last_fullsized":"2023-10-21T05:13:34.802849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'466538","reported_seq":930115,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:37.960772+0000","last_change":"2023-10-21T05:13:33.977983+0000","last_active":"2023-10-21T05:13:37.960772+0000","last_peered":"2023-10-21T05:13:37.960772+0000","last_clean":"2023-10-21T05:13:37.960772+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:13:37.960772+0000","last_undegraded":"2023-10-21T05:13:37.960772+0000","last_fullsized":"2023-10-21T05:13:37.960772+0000","mapping_epoch":16,"log_start":"87'456524","ondisk_log_start":"87'456524","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6762516499,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":464946,"num_read_kb":1829284,"num_write":464847,"num_write_kb":1859382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'16203","reported_seq":22373,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:37.889068+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:13:37.889068+0000","last_peered":"2023-10-21T05:13:37.889068+0000","last_clean":"2023-10-21T05:13:37.889068+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:13:37.889068+0000","last_undegraded":"2023-10-21T05:13:37.889068+0000","last_fullsized":"2023-10-21T05:13:37.889068+0000","mapping_epoch":13,"log_start":"80'6200","ondisk_log_start":"80'6200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":261,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-21T05:11:57.264170+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:11:57.264170+0000","last_peered":"2023-10-21T05:11:57.264170+0000","last_clean":"2023-10-21T05:11:57.264170+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:11:57.264170+0000","last_undegraded":"2023-10-21T05:11:57.264170+0000","last_fullsized":"2023-10-21T05:11:57.264170+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26995531795,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1863505,"num_read_kb":7333084,"num_write":1859304,"num_write_kb":7437210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15228604416,"data_stored":18122575910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40274,"ondisk_log_size":40274,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36913,"ondisk_log_size":36913,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739237,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8524608,"kb_used_data":7434552,"kb_used_omap":786,"kb_used_meta":1089261,"kb_avail":85847232,"statfs":{"total":96636764160,"available":87907565568,"internally_reserved":0,"allocated":7612981248,"data_stored":9059455911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":1115404027},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.45100000000000001},"min":{"1min":0.17899999999999999,"5min":0.153,"15min":0.153},"max":{"1min":2.6019999999999999,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":1.3839999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.54700000000000004,"15min":0.43099999999999999},"min":{"1min":0.22700000000000001,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":2.645,"5min":4.319,"15min":4.319},"last":1.208}]},{"osd":1,"last update":"Sat Oct 21 05:12:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.62,"15min":0.62},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.192},"max":{"1min":0.71199999999999997,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.53000000000000003,"15min":0.48699999999999999},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":4.8170000000000002,"15min":5.2199999999999998},"last":1.4350000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739234,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5349436,"kb_used_data":3764596,"kb_used_omap":353,"kb_used_meta":1584478,"kb_avail":89022404,"statfs":{"total":96636764160,"available":91158941696,"internally_reserved":0,"allocated":3854946304,"data_stored":4600851449,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1622505687},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,4,6,13,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:12:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.57199999999999995,"15min":0.54700000000000004},"min":{"1min":0.20300000000000001,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":0.67000000000000004,"5min":4.5599999999999996,"15min":6.8289999999999997},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.25900000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.69999999999999996,"5min":4.6109999999999998,"15min":6.8460000000000001},"last":0.30499999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:12:44 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.47399999999999998,"15min":0.45800000000000002},"min":{"1min":0.22600000000000001,"5min":0.20499999999999999,"15min":0.19800000000000001},"max":{"1min":8.7669999999999995,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38700000000000001,"15min":0.39600000000000002},"min":{"1min":0.26600000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.748,"5min":1.4590000000000001,"15min":4.306},"last":0.26900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739238,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5253024,"kb_used_data":3673496,"kb_used_omap":329,"kb_used_meta":1579190,"kb_avail":89118816,"statfs":{"total":96636764160,"available":91257667584,"internally_reserved":0,"allocated":3761659904,"data_stored":4462553034,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1617090570},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,7,7,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.371,"15min":0.39100000000000001},"min":{"1min":0.247,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":0.56799999999999995,"5min":1.7430000000000001,"15min":4.0199999999999996},"last":0.188},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.42299999999999999,"15min":0.44400000000000001},"min":{"1min":0.28599999999999998,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":0.60399999999999998,"5min":1.778,"15min":4.0919999999999996},"last":0.247}]},{"osd":2,"last update":"Sat Oct 21 05:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.49399999999999999,"15min":0.45700000000000002},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.214},"max":{"1min":2.411,"5min":6.2830000000000004,"15min":6.2830000000000004},"last":0.221},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57099999999999995,"15min":0.504},"min":{"1min":0.251,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.4849999999999999,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.32900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3761127424,"data_stored":4462063616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3854577664,"data_stored":4600627219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7612899328,"data_stored":9059885075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:13:43.847 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:43.847 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:43.848 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:43.848 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:13:43.849 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:03.850 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:14:04.495 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:14:04.496 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:14:04.528 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2265,"stamp":"2023-10-21T05:14:04.157535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27023821953,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1890636,"num_read_kb":7381373,"num_write":1868028,"num_write_kb":7471355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77284,"ondisk_log_size":77284,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19406300,"kb_used_data":14939904,"kb_used_omap":1470,"kb_used_meta":4464897,"kb_avail":263709220,"statfs":{"total":289910292480,"available":270038241280,"internally_reserved":0,"allocated":15298461696,"data_stored":18191734634,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1505316,"internal_metadata":4572055516},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,9,5,6,10,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15278080,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4546,"num_read_kb":18048,"num_write":4425,"num_write_kb":17700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002053"},"pg_stats":[{"pgid":"2.7","version":"89'13027","reported_seq":17908,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:14:00.393036+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:14:00.393036+0000","last_peered":"2023-10-21T05:14:00.393036+0000","last_clean":"2023-10-21T05:14:00.393036+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:14:00.393036+0000","last_undegraded":"2023-10-21T05:14:00.393036+0000","last_fullsized":"2023-10-21T05:14:00.393036+0000","mapping_epoch":13,"log_start":"72'3000","ondisk_log_start":"72'3000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"89'6646","reported_seq":10125,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:14:00.392544+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:14:00.392544+0000","last_peered":"2023-10-21T05:14:00.392544+0000","last_clean":"2023-10-21T05:14:00.392544+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:14:00.392544+0000","last_undegraded":"2023-10-21T05:14:00.392544+0000","last_fullsized":"2023-10-21T05:14:00.392544+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6646,"log_dups_size":0,"ondisk_log_size":6646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"89'3330","reported_seq":4708,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:57.911992+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:13:57.911992+0000","last_peered":"2023-10-21T05:13:57.911992+0000","last_clean":"2023-10-21T05:13:57.911992+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:13:57.911992+0000","last_undegraded":"2023-10-21T05:13:57.911992+0000","last_fullsized":"2023-10-21T05:13:57.911992+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3330,"log_dups_size":0,"ondisk_log_size":3330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'6593","reported_seq":9481,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:14:00.392510+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:14:00.392510+0000","last_peered":"2023-10-21T05:14:00.392510+0000","last_clean":"2023-10-21T05:14:00.392510+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:14:00.392510+0000","last_undegraded":"2023-10-21T05:14:00.392510+0000","last_fullsized":"2023-10-21T05:14:00.392510+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6593,"log_dups_size":0,"ondisk_log_size":6593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'476836","reported_seq":949754,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:01.628947+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:14:01.628947+0000","last_peered":"2023-10-21T05:14:01.628947+0000","last_clean":"2023-10-21T05:14:01.628947+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:14:01.628947+0000","last_undegraded":"2023-10-21T05:14:01.628947+0000","last_fullsized":"2023-10-21T05:14:01.628947+0000","mapping_epoch":16,"log_start":"88'466817","ondisk_log_start":"88'466817","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1498,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 381s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6844776448,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":474190,"num_read_kb":1866420,"num_write":474246,"num_write_kb":1896984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":216,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:44.846014+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:12:44.846014+0000","last_peered":"2023-10-21T05:12:44.846014+0000","last_clean":"2023-10-21T05:12:44.846014+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:12:44.846014+0000","last_undegraded":"2023-10-21T05:12:44.846014+0000","last_fullsized":"2023-10-21T05:12:44.846014+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'481096","reported_seq":959984,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:03.057554+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:14:03.057554+0000","last_peered":"2023-10-21T05:14:03.057554+0000","last_clean":"2023-10-21T05:14:03.057554+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:14:03.057554+0000","last_undegraded":"2023-10-21T05:14:03.057554+0000","last_fullsized":"2023-10-21T05:14:03.057554+0000","mapping_epoch":16,"log_start":"88'471018","ondisk_log_start":"88'471018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1463,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 372s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6960029696,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":480236,"num_read_kb":1889924,"num_write":478450,"num_write_kb":1913800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3806,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:14:00.543453+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:14:00.543453+0000","last_peered":"2023-10-21T05:14:00.543453+0000","last_clean":"2023-10-21T05:14:00.543453+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:14:00.543453+0000","last_undegraded":"2023-10-21T05:14:00.543453+0000","last_fullsized":"2023-10-21T05:14:00.543453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'450362","reported_seq":899150,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:01.597764+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:14:01.597764+0000","last_peered":"2023-10-21T05:14:01.597764+0000","last_clean":"2023-10-21T05:14:01.597764+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:14:01.597764+0000","last_undegraded":"2023-10-21T05:14:01.597764+0000","last_fullsized":"2023-10-21T05:14:01.597764+0000","mapping_epoch":16,"log_start":"88'440319","ondisk_log_start":"88'440319","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2021,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 385s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6448390144,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":449899,"num_read_kb":1770404,"num_write":447589,"num_write_kb":1790356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3742,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:14:00.392934+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:14:00.392934+0000","last_peered":"2023-10-21T05:14:00.392934+0000","last_clean":"2023-10-21T05:14:00.392934+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:14:00.392934+0000","last_undegraded":"2023-10-21T05:14:00.392934+0000","last_fullsized":"2023-10-21T05:14:00.392934+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'468696","reported_seq":934560,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:14:03.087551+0000","last_change":"2023-10-21T05:14:02.733264+0000","last_active":"2023-10-21T05:14:03.087551+0000","last_peered":"2023-10-21T05:14:03.087551+0000","last_clean":"2023-10-21T05:14:03.087551+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:14:03.087551+0000","last_undegraded":"2023-10-21T05:14:03.087551+0000","last_fullsized":"2023-10-21T05:14:03.087551+0000","mapping_epoch":16,"log_start":"88'458624","ondisk_log_start":"88'458624","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6770032659,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":467223,"num_read_kb":1838304,"num_write":467005,"num_write_kb":1868014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'16259","reported_seq":22454,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:14:00.393478+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:14:00.393478+0000","last_peered":"2023-10-21T05:14:00.393478+0000","last_clean":"2023-10-21T05:14:00.393478+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:14:00.393478+0000","last_undegraded":"2023-10-21T05:14:00.393478+0000","last_fullsized":"2023-10-21T05:14:00.393478+0000","mapping_epoch":13,"log_start":"80'6200","ondisk_log_start":"80'6200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":262,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:44.848162+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:12:44.848162+0000","last_peered":"2023-10-21T05:12:44.848162+0000","last_clean":"2023-10-21T05:12:44.848162+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:12:44.848162+0000","last_undegraded":"2023-10-21T05:12:44.848162+0000","last_fullsized":"2023-10-21T05:12:44.848162+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27023228947,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1871548,"num_read_kb":7365052,"num_write":1867290,"num_write_kb":7469154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15297548288,"data_stored":18191519782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40212,"ondisk_log_size":40212,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":36973,"ondisk_log_size":36973,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739242,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8678680,"kb_used_data":7469516,"kb_used_omap":786,"kb_used_meta":1208365,"kb_avail":85693160,"statfs":{"total":96636764160,"available":87749795840,"internally_reserved":0,"allocated":7648784384,"data_stored":9095259047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":1237366523},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.61199999999999999,"15min":0.47199999999999998},"min":{"1min":0.20599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":4.8540000000000001,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.63900000000000001,"15min":0.46600000000000003},"min":{"1min":0.192,"5min":0.192,"15min":0.185},"max":{"1min":8.8510000000000009,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.26600000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:13:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.59099999999999997,"15min":0.61799999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.75700000000000001,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":1.238},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.56100000000000005,"15min":0.50900000000000001},"min":{"1min":0.245,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.8979999999999997,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.45200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739239,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5428260,"kb_used_data":3782492,"kb_used_omap":353,"kb_used_meta":1645406,"kb_avail":88943580,"statfs":{"total":96636764160,"available":91078225920,"internally_reserved":0,"allocated":3873271808,"data_stored":4619176953,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1684895959},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,6,1,3,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.53700000000000003,"15min":0.52300000000000002},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":2.0609999999999999,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.52200000000000002,"15min":0.495},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.186},"max":{"1min":2.0270000000000001,"5min":4.6109999999999998,"15min":4.6109999999999998},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:13:48 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.44800000000000001,"15min":0.443},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47999999999999998,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.374,"15min":0.39200000000000002},"min":{"1min":0.26900000000000002,"5min":0.221,"15min":0.20499999999999999},"max":{"1min":0.41499999999999998,"5min":0.77900000000000003,"15min":4.306},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739242,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5299360,"kb_used_data":3687896,"kb_used_omap":329,"kb_used_meta":1611126,"kb_avail":89072480,"statfs":{"total":96636764160,"available":91210219520,"internally_reserved":0,"allocated":3776405504,"data_stored":4477298634,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1649793034},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.373,"15min":0.38700000000000001},"min":{"1min":0.188,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":0.93300000000000005,"5min":1.7430000000000001,"15min":4.0199999999999996},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.42699999999999999,"15min":0.442},"min":{"1min":0.247,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":0.97799999999999998,"5min":1.778,"15min":4.0919999999999996},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.56799999999999995,"15min":0.47799999999999998},"min":{"1min":0.221,"5min":0.22,"15min":0.214},"max":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":6.7430000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.57999999999999996,"15min":0.504},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":1.2909999999999999,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.32200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3775836160,"data_stored":4476772352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3872600064,"data_stored":4618649619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7649112064,"data_stored":9096097811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:14:04.532 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:04.532 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:04.533 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:04.534 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:04.534 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:24.536 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:14:25.147 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:14:25.149 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:14:25.177 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2276,"stamp":"2023-10-21T05:14:24.160982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27048221825,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1897493,"num_read_kb":7408641,"num_write":1875054,"num_write_kb":7499459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77409,"ondisk_log_size":77409,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19620316,"kb_used_data":14995780,"kb_used_omap":1470,"kb_used_meta":4623041,"kb_avail":263495204,"statfs":{"total":289910292480,"available":269819088896,"internally_reserved":0,"allocated":15355678720,"data_stored":18248938455,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1505316,"internal_metadata":4733994972},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,6,6,34,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15306752,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4265,"num_read_kb":16960,"num_write":4398,"num_write_kb":17592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002084"},"pg_stats":[{"pgid":"2.7","version":"90'13092","reported_seq":17990,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:22.876641+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:14:22.876641+0000","last_peered":"2023-10-21T05:14:22.876641+0000","last_clean":"2023-10-21T05:14:22.876641+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:14:22.876641+0000","last_undegraded":"2023-10-21T05:14:22.876641+0000","last_fullsized":"2023-10-21T05:14:22.876641+0000","mapping_epoch":13,"log_start":"72'3000","ondisk_log_start":"72'3000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"90'6679","reported_seq":10167,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:22.856710+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:14:22.856710+0000","last_peered":"2023-10-21T05:14:22.856710+0000","last_clean":"2023-10-21T05:14:22.856710+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:14:22.856710+0000","last_undegraded":"2023-10-21T05:14:22.856710+0000","last_fullsized":"2023-10-21T05:14:22.856710+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6679,"log_dups_size":0,"ondisk_log_size":6679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"90'3347","reported_seq":4730,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:22.844172+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:14:22.844172+0000","last_peered":"2023-10-21T05:14:22.844172+0000","last_clean":"2023-10-21T05:14:22.844172+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:14:22.844172+0000","last_undegraded":"2023-10-21T05:14:22.844172+0000","last_fullsized":"2023-10-21T05:14:22.844172+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3347,"log_dups_size":0,"ondisk_log_size":3347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"90'6626","reported_seq":9523,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:22.857335+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:14:22.857335+0000","last_peered":"2023-10-21T05:14:22.857335+0000","last_clean":"2023-10-21T05:14:22.857335+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:14:22.857335+0000","last_undegraded":"2023-10-21T05:14:22.857335+0000","last_fullsized":"2023-10-21T05:14:22.857335+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6626,"log_dups_size":0,"ondisk_log_size":6626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"90'478705","reported_seq":953410,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:21.738182+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:14:21.738182+0000","last_peered":"2023-10-21T05:14:21.738182+0000","last_clean":"2023-10-21T05:14:21.738182+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:14:21.738182+0000","last_undegraded":"2023-10-21T05:14:21.738182+0000","last_fullsized":"2023-10-21T05:14:21.738182+0000","mapping_epoch":16,"log_start":"88'468617","ondisk_log_start":"88'468617","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1564,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 401s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6850936832,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":475965,"num_read_kb":1873480,"num_write":476075,"num_write_kb":1904300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":217,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:51.178363+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:12:51.178363+0000","last_peered":"2023-10-21T05:12:51.178363+0000","last_clean":"2023-10-21T05:12:51.178363+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:12:51.178363+0000","last_undegraded":"2023-10-21T05:12:51.178363+0000","last_fullsized":"2023-10-21T05:12:51.178363+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"90'482933","reported_seq":963567,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:23.157557+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:14:23.157557+0000","last_peered":"2023-10-21T05:14:23.157557+0000","last_clean":"2023-10-21T05:14:23.157557+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:14:23.157557+0000","last_undegraded":"2023-10-21T05:14:23.157557+0000","last_fullsized":"2023-10-21T05:14:23.157557+0000","mapping_epoch":16,"log_start":"88'472918","ondisk_log_start":"88'472918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1543,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 392s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6966415360,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":481972,"num_read_kb":1896832,"num_write":480238,"num_write_kb":1920952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3819,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:20.926344+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:14:20.926344+0000","last_peered":"2023-10-21T05:14:20.926344+0000","last_clean":"2023-10-21T05:14:20.926344+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:14:20.926344+0000","last_undegraded":"2023-10-21T05:14:20.926344+0000","last_fullsized":"2023-10-21T05:14:20.926344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"90'452135","reported_seq":902607,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:21.719731+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:14:21.719731+0000","last_peered":"2023-10-21T05:14:21.719731+0000","last_clean":"2023-10-21T05:14:21.719731+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:14:21.719731+0000","last_undegraded":"2023-10-21T05:14:21.719731+0000","last_fullsized":"2023-10-21T05:14:21.719731+0000","mapping_epoch":16,"log_start":"88'442119","ondisk_log_start":"88'442119","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2153,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 405s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6454149120,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":451568,"num_read_kb":1777036,"num_write":449281,"num_write_kb":1797124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3755,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:20.882213+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:14:20.882213+0000","last_peered":"2023-10-21T05:14:20.882213+0000","last_clean":"2023-10-21T05:14:20.882213+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:14:20.882213+0000","last_undegraded":"2023-10-21T05:14:20.882213+0000","last_fullsized":"2023-10-21T05:14:20.882213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"90'470413","reported_seq":937967,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:23.150790+0000","last_change":"2023-10-21T05:14:22.585137+0000","last_active":"2023-10-21T05:14:23.150790+0000","last_peered":"2023-10-21T05:14:23.150790+0000","last_clean":"2023-10-21T05:14:23.150790+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:14:23.150790+0000","last_undegraded":"2023-10-21T05:14:23.150790+0000","last_fullsized":"2023-10-21T05:14:23.150790+0000","mapping_epoch":16,"log_start":"88'460324","ondisk_log_start":"88'460324","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6776127507,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":468900,"num_read_kb":1844972,"num_write":468722,"num_write_kb":1874882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"90'16340","reported_seq":22556,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:22.936351+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:14:22.936351+0000","last_peered":"2023-10-21T05:14:22.936351+0000","last_clean":"2023-10-21T05:14:22.936351+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:14:22.936351+0000","last_undegraded":"2023-10-21T05:14:22.936351+0000","last_fullsized":"2023-10-21T05:14:22.936351+0000","mapping_epoch":13,"log_start":"80'6300","ondisk_log_start":"80'6300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":263,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:58.045147+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:13:58.045147+0000","last_peered":"2023-10-21T05:13:58.045147+0000","last_clean":"2023-10-21T05:13:58.045147+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:13:58.045147+0000","last_undegraded":"2023-10-21T05:13:58.045147+0000","last_fullsized":"2023-10-21T05:13:58.045147+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27047628819,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1878405,"num_read_kb":7392320,"num_write":1874316,"num_write_kb":7497258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15355203584,"data_stored":18249175078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40208,"ondisk_log_size":40208,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37102,"ondisk_log_size":37102,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739246,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8785768,"kb_used_data":7497568,"kb_used_omap":786,"kb_used_meta":1287405,"kb_avail":85586072,"statfs":{"total":96636764160,"available":87640137728,"internally_reserved":0,"allocated":7677509632,"data_stored":9123979894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":1318303483},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.61199999999999999,"15min":0.47199999999999998},"min":{"1min":0.20599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":4.8540000000000001,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.63900000000000001,"15min":0.46600000000000003},"min":{"1min":0.192,"5min":0.192,"15min":0.185},"max":{"1min":8.8510000000000009,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.436}]},{"osd":1,"last update":"Sat Oct 21 05:13:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.59099999999999997,"15min":0.61799999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.75700000000000001,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.56100000000000005,"15min":0.50900000000000001},"min":{"1min":0.245,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.8979999999999997,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.53600000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739243,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5478360,"kb_used_data":3796496,"kb_used_omap":353,"kb_used_meta":1681502,"kb_avail":88893480,"statfs":{"total":96636764160,"available":91026923520,"internally_reserved":0,"allocated":3887611904,"data_stored":4633512648,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1721858263},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.53700000000000003,"15min":0.52300000000000002},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":2.0609999999999999,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.52200000000000002,"15min":0.495},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.186},"max":{"1min":2.0270000000000001,"5min":4.6109999999999998,"15min":4.6109999999999998},"last":0.34699999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:13:48 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.44800000000000001,"15min":0.443},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47999999999999998,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.433},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.374,"15min":0.39200000000000002},"min":{"1min":0.26900000000000002,"5min":0.221,"15min":0.20499999999999999},"max":{"1min":0.41499999999999998,"5min":0.77900000000000003,"15min":4.306},"last":0.38100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739246,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5356188,"kb_used_data":3701716,"kb_used_omap":329,"kb_used_meta":1654134,"kb_avail":89015652,"statfs":{"total":96636764160,"available":91152027648,"internally_reserved":0,"allocated":3790557184,"data_stored":4491445913,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1693833226},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,0,29],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.373,"15min":0.38700000000000001},"min":{"1min":0.188,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":0.93300000000000005,"5min":1.7430000000000001,"15min":4.0199999999999996},"last":0.505},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.42699999999999999,"15min":0.442},"min":{"1min":0.247,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":0.97799999999999998,"5min":1.778,"15min":4.0919999999999996},"last":0.54800000000000004}]},{"osd":2,"last update":"Sat Oct 21 05:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.56799999999999995,"15min":0.47799999999999998},"min":{"1min":0.221,"5min":0.22,"15min":0.214},"max":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.57999999999999996,"15min":0.504},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":1.2909999999999999,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.60199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3790270464,"data_stored":4491206656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3886940160,"data_stored":4632989715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7677992960,"data_stored":9124978707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:14:25.179 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:25.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:25.180 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:25.181 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:25.181 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:14:25.716 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-21T05:14:25.756 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:14:26.235 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:14:26.263 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:14:26.693 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:14:26.682+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:14:26.788 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-21T05:14:26.817 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:14:27.437 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:14:27.483 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:47.485 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:14:47.918 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T05:14:47.914+0000 7f3b98b88640 0 --2- 172.21.15.133:0/1512880394 >> v2:172.21.15.133:3300/0 conn(0x7f3b94128a90 0x7f3b94128ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-21T05:14:48.438 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:14:48.439 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:14:48.497 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2288,"stamp":"2023-10-21T05:14:48.165025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27075513473,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1905017,"num_read_kb":7438577,"num_write":1882933,"num_write_kb":7530975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77396,"ondisk_log_size":77396,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19899504,"kb_used_data":15062488,"kb_used_omap":1470,"kb_used_meta":4835521,"kb_avail":263216016,"statfs":{"total":289910292480,"available":269533200384,"internally_reserved":0,"allocated":15423987712,"data_stored":18317247447,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1505316,"internal_metadata":4951574492},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,7,5,25,22,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15331328,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4151,"num_read_kb":16516,"num_write":4427,"num_write_kb":17708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002049"},"pg_stats":[{"pgid":"2.7","version":"90'13149","reported_seq":18063,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:43.249124+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:14:43.249124+0000","last_peered":"2023-10-21T05:14:43.249124+0000","last_clean":"2023-10-21T05:14:43.249124+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:14:43.249124+0000","last_undegraded":"2023-10-21T05:14:43.249124+0000","last_fullsized":"2023-10-21T05:14:43.249124+0000","mapping_epoch":13,"log_start":"72'3100","ondisk_log_start":"72'3100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"90'6708","reported_seq":10204,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:43.209138+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:14:43.209138+0000","last_peered":"2023-10-21T05:14:43.209138+0000","last_clean":"2023-10-21T05:14:43.209138+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:14:43.209138+0000","last_undegraded":"2023-10-21T05:14:43.209138+0000","last_fullsized":"2023-10-21T05:14:43.209138+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6708,"log_dups_size":0,"ondisk_log_size":6708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"90'3361","reported_seq":4748,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:41.357124+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:14:41.357124+0000","last_peered":"2023-10-21T05:14:41.357124+0000","last_clean":"2023-10-21T05:14:41.357124+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:14:41.357124+0000","last_undegraded":"2023-10-21T05:14:41.357124+0000","last_fullsized":"2023-10-21T05:14:41.357124+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3361,"log_dups_size":0,"ondisk_log_size":3361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"90'6655","reported_seq":9560,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:43.209753+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:14:43.209753+0000","last_peered":"2023-10-21T05:14:43.209753+0000","last_clean":"2023-10-21T05:14:43.209753+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:14:43.209753+0000","last_undegraded":"2023-10-21T05:14:43.209753+0000","last_fullsized":"2023-10-21T05:14:43.209753+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6655,"log_dups_size":0,"ondisk_log_size":6655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"90'481039","reported_seq":957845,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:46.973773+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:14:46.973773+0000","last_peered":"2023-10-21T05:14:46.973773+0000","last_clean":"2023-10-21T05:14:46.973773+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:14:46.973773+0000","last_undegraded":"2023-10-21T05:14:46.973773+0000","last_fullsized":"2023-10-21T05:14:46.973773+0000","mapping_epoch":16,"log_start":"89'471017","ondisk_log_start":"89'471017","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1693,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 427s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6858756096,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":478052,"num_read_kb":1881780,"num_write":478331,"num_write_kb":1913324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":217,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:12:51.178363+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:12:51.178363+0000","last_peered":"2023-10-21T05:12:51.178363+0000","last_clean":"2023-10-21T05:12:51.178363+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:12:51.178363+0000","last_undegraded":"2023-10-21T05:12:51.178363+0000","last_fullsized":"2023-10-21T05:12:51.178363+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"90'484783","reported_seq":967159,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:43.290341+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:14:43.290341+0000","last_peered":"2023-10-21T05:14:43.290341+0000","last_clean":"2023-10-21T05:14:43.290341+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:14:43.290341+0000","last_undegraded":"2023-10-21T05:14:43.290341+0000","last_fullsized":"2023-10-21T05:14:43.290341+0000","mapping_epoch":16,"log_start":"89'474718","ondisk_log_start":"89'474718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1598,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 412s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6972829696,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":483703,"num_read_kb":1903720,"num_write":482049,"num_write_kb":1928196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3835,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:46.497601+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:14:46.497601+0000","last_peered":"2023-10-21T05:14:46.497601+0000","last_clean":"2023-10-21T05:14:46.497601+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:14:46.497601+0000","last_undegraded":"2023-10-21T05:14:46.497601+0000","last_fullsized":"2023-10-21T05:14:46.497601+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"90'454336","reported_seq":906871,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:14:46.956067+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:14:46.956067+0000","last_peered":"2023-10-21T05:14:46.956067+0000","last_clean":"2023-10-21T05:14:46.956067+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:14:46.956067+0000","last_undegraded":"2023-10-21T05:14:46.956067+0000","last_fullsized":"2023-10-21T05:14:46.956067+0000","mapping_epoch":16,"log_start":"89'444320","ondisk_log_start":"89'444320","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2301,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 431s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6461239296,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":453615,"num_read_kb":1785188,"num_write":451389,"num_write_kb":1805556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3770,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:46.483620+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:14:46.483620+0000","last_peered":"2023-10-21T05:14:46.483620+0000","last_clean":"2023-10-21T05:14:46.483620+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:14:46.483620+0000","last_undegraded":"2023-10-21T05:14:46.483620+0000","last_fullsized":"2023-10-21T05:14:46.483620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"90'472117","reported_seq":941337,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:43.294697+0000","last_change":"2023-10-21T05:14:40.855505+0000","last_active":"2023-10-21T05:14:43.294697+0000","last_peered":"2023-10-21T05:14:43.294697+0000","last_clean":"2023-10-21T05:14:43.294697+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:14:43.294697+0000","last_undegraded":"2023-10-21T05:14:43.294697+0000","last_fullsized":"2023-10-21T05:14:43.294697+0000","mapping_epoch":16,"log_start":"89'462024","ondisk_log_start":"89'462024","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6782095379,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":470559,"num_read_kb":1851568,"num_write":470426,"num_write_kb":1881698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"90'16410","reported_seq":22646,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:43.263447+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:14:43.263447+0000","last_peered":"2023-10-21T05:14:43.263447+0000","last_clean":"2023-10-21T05:14:43.263447+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:14:43.263447+0000","last_undegraded":"2023-10-21T05:14:43.263447+0000","last_fullsized":"2023-10-21T05:14:43.263447+0000","mapping_epoch":13,"log_start":"80'6400","ondisk_log_start":"80'6400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":263,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:58.045147+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:13:58.045147+0000","last_peered":"2023-10-21T05:13:58.045147+0000","last_clean":"2023-10-21T05:13:58.045147+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:13:58.045147+0000","last_undegraded":"2023-10-21T05:13:58.045147+0000","last_fullsized":"2023-10-21T05:13:58.045147+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27074920467,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1885929,"num_read_kb":7422256,"num_write":1882195,"num_write_kb":7528774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15423315968,"data_stored":18317287462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40196,"ondisk_log_size":40196,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37101,"ondisk_log_size":37101,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739251,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8911180,"kb_used_data":7532740,"kb_used_omap":786,"kb_used_meta":1377645,"kb_avail":85460660,"statfs":{"total":96636764160,"available":87511715840,"internally_reserved":0,"allocated":7713525760,"data_stored":9159996022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":1410709243},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.61199999999999999,"15min":0.47199999999999998},"min":{"1min":0.20599999999999999,"5min":0.153,"15min":0.153},"max":{"1min":4.8540000000000001,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.63900000000000001,"15min":0.46600000000000003},"min":{"1min":0.192,"5min":0.192,"15min":0.185},"max":{"1min":8.8510000000000009,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:13:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.59099999999999997,"15min":0.61799999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.75700000000000001,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.56100000000000005,"15min":0.50900000000000001},"min":{"1min":0.245,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.8979999999999997,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.32900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739247,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5553668,"kb_used_data":3810620,"kb_used_omap":353,"kb_used_meta":1742686,"kb_avail":88818172,"statfs":{"total":96636764160,"available":90949808128,"internally_reserved":0,"allocated":3902074880,"data_stored":4647975624,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1784510679},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,10,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:13:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.53700000000000003,"15min":0.52300000000000002},"min":{"1min":0.21199999999999999,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":2.0609999999999999,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.378},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.52200000000000002,"15min":0.495},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.186},"max":{"1min":2.0270000000000001,"5min":4.6109999999999998,"15min":4.6109999999999998},"last":0.40400000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:13:48 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.44800000000000001,"15min":0.443},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47999999999999998,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.374,"15min":0.39200000000000002},"min":{"1min":0.26900000000000002,"5min":0.221,"15min":0.20499999999999999},"max":{"1min":0.41499999999999998,"5min":0.77900000000000003,"15min":4.306},"last":0.35599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739251,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5434656,"kb_used_data":3719128,"kb_used_omap":329,"kb_used_meta":1715190,"kb_avail":88937184,"statfs":{"total":96636764160,"available":91071676416,"internally_reserved":0,"allocated":3808387072,"data_stored":4509275801,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1756354570},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,2,13,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.373,"15min":0.38700000000000001},"min":{"1min":0.188,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":0.93300000000000005,"5min":1.7430000000000001,"15min":4.0199999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.42699999999999999,"15min":0.442},"min":{"1min":0.247,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":0.97799999999999998,"5min":1.778,"15min":4.0919999999999996},"last":0.42599999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.56799999999999995,"15min":0.47799999999999998},"min":{"1min":0.221,"5min":0.22,"15min":0.214},"max":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.57999999999999996,"15min":0.504},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":1.2909999999999999,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3808145408,"data_stored":4509081600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3901341696,"data_stored":4647391251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7713828864,"data_stored":9160814611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:14:48.499 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:48.499 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:48.499 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:48.500 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:14:48.500 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:14:52.671 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:14:52.666+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:14:52.675 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:14:52.670+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:14:52.700 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:14:52.694+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:14:52.700 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:14:52.694+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:14:52.700 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:14:52.694+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:15:03.499 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T05:15:03.499 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:15:03.500 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:15:03.500 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:15:03.500 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:15:03.500 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T05:15:03.501 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:15:03.501 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:15:03.501 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:15:03.502 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:15:03.502 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:15:03.502 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:15:03.502 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:15:03.503 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:03.494+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:15:04.751 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:04.746+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:15:04.761 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:04.754+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:15:04.761 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:04.754+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:15:08.501 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:15:09.033 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:15:09.034 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:15:09.087 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2298,"stamp":"2023-10-21T05:15:08.168400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27099929729,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1911807,"num_read_kb":7465613,"num_write":1889998,"num_write_kb":7559235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77375,"ondisk_log_size":77375,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20104784,"kb_used_data":15112504,"kb_used_omap":1470,"kb_used_meta":4990785,"kb_avail":263010736,"statfs":{"total":289910292480,"available":269322993664,"internally_reserved":0,"allocated":15475204096,"data_stored":18368463831,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1505316,"internal_metadata":5110564828},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,12,13,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15024128,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4232,"num_read_kb":16852,"num_write":4385,"num_write_kb":17540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002024"},"pg_stats":[{"pgid":"2.7","version":"90'13179","reported_seq":18107,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:15:03.491190+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:15:03.491190+0000","last_peered":"2023-10-21T05:15:03.491190+0000","last_clean":"2023-10-21T05:15:03.491190+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:15:03.491190+0000","last_undegraded":"2023-10-21T05:15:03.491190+0000","last_fullsized":"2023-10-21T05:15:03.491190+0000","mapping_epoch":13,"log_start":"72'3100","ondisk_log_start":"72'3100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"90'6723","reported_seq":10227,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:15:03.464065+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:15:03.464065+0000","last_peered":"2023-10-21T05:15:03.464065+0000","last_clean":"2023-10-21T05:15:03.464065+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:15:03.464065+0000","last_undegraded":"2023-10-21T05:15:03.464065+0000","last_fullsized":"2023-10-21T05:15:03.464065+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6723,"log_dups_size":0,"ondisk_log_size":6723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"90'3369","reported_seq":4761,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:15:01.832966+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:15:01.832966+0000","last_peered":"2023-10-21T05:15:01.832966+0000","last_clean":"2023-10-21T05:15:01.832966+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:15:01.832966+0000","last_undegraded":"2023-10-21T05:15:01.832966+0000","last_fullsized":"2023-10-21T05:15:01.832966+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3369,"log_dups_size":0,"ondisk_log_size":3369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"90'6671","reported_seq":9585,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:15:03.472633+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:15:03.472633+0000","last_peered":"2023-10-21T05:15:03.472633+0000","last_clean":"2023-10-21T05:15:03.472633+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:15:03.472633+0000","last_undegraded":"2023-10-21T05:15:03.472633+0000","last_fullsized":"2023-10-21T05:15:03.472633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6671,"log_dups_size":0,"ondisk_log_size":6671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"90'482931","reported_seq":961455,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:07.081423+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:15:07.081423+0000","last_peered":"2023-10-21T05:15:07.081423+0000","last_clean":"2023-10-21T05:15:07.081423+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:15:07.081423+0000","last_undegraded":"2023-10-21T05:15:07.081423+0000","last_fullsized":"2023-10-21T05:15:07.081423+0000","mapping_epoch":16,"log_start":"89'472917","ondisk_log_start":"89'472917","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1805,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 447s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6864977920,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":479757,"num_read_kb":1888560,"num_write":480150,"num_write_kb":1920600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":218,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:04.870621+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:14:04.870621+0000","last_peered":"2023-10-21T05:14:04.870621+0000","last_clean":"2023-10-21T05:14:04.870621+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:14:04.870621+0000","last_undegraded":"2023-10-21T05:14:04.870621+0000","last_fullsized":"2023-10-21T05:14:04.870621+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"90'486636","reported_seq":970751,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:03.483578+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:15:03.483578+0000","last_peered":"2023-10-21T05:15:03.483578+0000","last_clean":"2023-10-21T05:15:03.483578+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:15:03.483578+0000","last_undegraded":"2023-10-21T05:15:03.483578+0000","last_fullsized":"2023-10-21T05:15:03.483578+0000","mapping_epoch":16,"log_start":"89'476618","ondisk_log_start":"89'476618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1718,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 432s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6979149824,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":485443,"num_read_kb":1910652,"num_write":483824,"num_write_kb":1935296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3844,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:15:01.832421+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:15:01.832421+0000","last_peered":"2023-10-21T05:15:01.832421+0000","last_clean":"2023-10-21T05:15:01.832421+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:15:01.832421+0000","last_undegraded":"2023-10-21T05:15:01.832421+0000","last_fullsized":"2023-10-21T05:15:01.832421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"90'456134","reported_seq":910326,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:07.086975+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:15:07.086975+0000","last_peered":"2023-10-21T05:15:07.086975+0000","last_clean":"2023-10-21T05:15:07.086975+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:15:07.086975+0000","last_undegraded":"2023-10-21T05:15:07.086975+0000","last_fullsized":"2023-10-21T05:15:07.086975+0000","mapping_epoch":16,"log_start":"89'446120","ondisk_log_start":"89'446120","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2405,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 451s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6467022848,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":455259,"num_read_kb":1791720,"num_write":453124,"num_write_kb":1812496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3779,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:15:01.832273+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:15:01.832273+0000","last_peered":"2023-10-21T05:15:01.832273+0000","last_clean":"2023-10-21T05:15:01.832273+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:15:01.832273+0000","last_undegraded":"2023-10-21T05:15:01.832273+0000","last_fullsized":"2023-10-21T05:15:01.832273+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"90'473853","reported_seq":944787,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:15:03.449924+0000","last_change":"2023-10-21T05:14:57.108002+0000","last_active":"2023-10-21T05:15:03.449924+0000","last_peered":"2023-10-21T05:15:03.449924+0000","last_clean":"2023-10-21T05:15:03.449924+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:15:03.449924+0000","last_undegraded":"2023-10-21T05:15:03.449924+0000","last_fullsized":"2023-10-21T05:15:03.449924+0000","mapping_epoch":16,"log_start":"89'463824","ondisk_log_start":"89'463824","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6788186131,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":472260,"num_read_kb":1858360,"num_write":472162,"num_write_kb":1888642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"90'16441","reported_seq":22698,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:15:01.832536+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:15:01.832536+0000","last_peered":"2023-10-21T05:15:01.832536+0000","last_clean":"2023-10-21T05:15:01.832536+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:15:01.832536+0000","last_undegraded":"2023-10-21T05:15:01.832536+0000","last_fullsized":"2023-10-21T05:15:01.832536+0000","mapping_epoch":13,"log_start":"80'6400","ondisk_log_start":"80'6400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":263,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-21T05:13:58.045147+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:13:58.045147+0000","last_peered":"2023-10-21T05:13:58.045147+0000","last_clean":"2023-10-21T05:13:58.045147+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:13:58.045147+0000","last_undegraded":"2023-10-21T05:13:58.045147+0000","last_fullsized":"2023-10-21T05:13:58.045147+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27099336723,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1892719,"num_read_kb":7449292,"num_write":1889260,"num_write_kb":7557034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15473975296,"data_stored":18367946790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40075,"ondisk_log_size":40075,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37201,"ondisk_log_size":37201,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739254,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9019868,"kb_used_data":7554196,"kb_used_omap":786,"kb_used_meta":1464877,"kb_avail":85351972,"statfs":{"total":96636764160,"available":87400419328,"internally_reserved":0,"allocated":7735496704,"data_stored":9181966966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":1500034811},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.56499999999999995,"15min":0.46700000000000003},"min":{"1min":0.184,"5min":0.17899999999999999,"15min":0.153},"max":{"1min":0.52000000000000002,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.55500000000000005,"15min":0.45900000000000002},"min":{"1min":0.252,"5min":0.192,"15min":0.185},"max":{"1min":0.51200000000000001,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.32100000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:14:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.55900000000000005,"15min":0.621},"min":{"1min":0.224,"5min":0.188,"15min":0.188},"max":{"1min":1.238,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.52200000000000002,"15min":0.51200000000000001},"min":{"1min":0.26800000000000002,"5min":0.23200000000000001,"15min":0.20999999999999999},"max":{"1min":0.63300000000000001,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.19800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739251,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5599828,"kb_used_data":3824844,"kb_used_omap":353,"kb_used_meta":1774622,"kb_avail":88772012,"statfs":{"total":96636764160,"available":90902540288,"internally_reserved":0,"allocated":3916640256,"data_stored":4662541000,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1817213143},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:53 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.503,"15min":0.51700000000000002},"min":{"1min":0.26400000000000001,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":0.91900000000000004,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":0.61099999999999999,"5min":4.6109999999999998,"15min":4.6109999999999998},"last":0.29699999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:14:53 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.45500000000000002,"15min":0.44400000000000001},"min":{"1min":0.221,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.95499999999999996,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.31},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.38100000000000001,"15min":0.39400000000000002},"min":{"1min":0.247,"5min":0.221,"15min":0.20499999999999999},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":4.306},"last":0.32900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739255,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5485088,"kb_used_data":3733464,"kb_used_omap":329,"kb_used_meta":1751286,"kb_avail":88886752,"statfs":{"total":96636764160,"available":91020034048,"internally_reserved":0,"allocated":3823067136,"data_stored":4523955865,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1793316874},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,8,11,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:14:58 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.36899999999999999,"15min":0.375},"min":{"1min":0.22500000000000001,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":0.505,"5min":1.7430000000000001,"15min":3.7370000000000001},"last":5.0099999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42699999999999999,"15min":0.42999999999999999},"min":{"1min":0.312,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":0.54800000000000004,"5min":1.778,"15min":3.7709999999999999},"last":5.0469999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:14:58 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.55800000000000005,"15min":0.46000000000000002},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.214},"max":{"1min":0.57699999999999996,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61199999999999999,"15min":0.497},"min":{"1min":0.252,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":3.7799999999999998,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3822706688,"data_stored":4523642880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3915718656,"data_stored":4661768211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7735549952,"data_stored":9182535699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:15:09.089 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:09.089 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:09.090 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:09.090 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:09.091 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:15:29.092 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:15:29.649 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:15:29.649 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:15:29.711 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2309,"stamp":"2023-10-21T05:15:28.171980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27125075073,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1918414,"num_read_kb":7491993,"num_write":1897225,"num_write_kb":7588143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77401,"ondisk_log_size":77401,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20333584,"kb_used_data":15169144,"kb_used_omap":1470,"kb_used_meta":5162945,"kb_avail":262781936,"statfs":{"total":289910292480,"available":269088702464,"internally_reserved":0,"allocated":15533203456,"data_stored":18426449859,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1505316,"internal_metadata":5286856668},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,0,6,33,11,17,8,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15659008,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4165,"num_read_kb":16640,"num_write":4506,"num_write_kb":18024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002206"},"pg_stats":[{"pgid":"2.7","version":"91'13232","reported_seq":18179,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:22.312777+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:15:22.312777+0000","last_peered":"2023-10-21T05:15:22.312777+0000","last_clean":"2023-10-21T05:15:22.312777+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:15:22.312777+0000","last_undegraded":"2023-10-21T05:15:22.312777+0000","last_fullsized":"2023-10-21T05:15:22.312777+0000","mapping_epoch":13,"log_start":"74'3200","ondisk_log_start":"74'3200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"91'6750","reported_seq":10263,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:22.312067+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:15:22.312067+0000","last_peered":"2023-10-21T05:15:22.312067+0000","last_clean":"2023-10-21T05:15:22.312067+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:15:22.312067+0000","last_undegraded":"2023-10-21T05:15:22.312067+0000","last_fullsized":"2023-10-21T05:15:22.312067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6750,"log_dups_size":0,"ondisk_log_size":6750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"91'3383","reported_seq":4780,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:21.278973+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:15:21.278973+0000","last_peered":"2023-10-21T05:15:21.278973+0000","last_clean":"2023-10-21T05:15:21.278973+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:15:21.278973+0000","last_undegraded":"2023-10-21T05:15:21.278973+0000","last_fullsized":"2023-10-21T05:15:21.278973+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3383,"log_dups_size":0,"ondisk_log_size":3383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"91'6698","reported_seq":9621,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:22.312073+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:15:22.312073+0000","last_peered":"2023-10-21T05:15:22.312073+0000","last_clean":"2023-10-21T05:15:22.312073+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:15:22.312073+0000","last_undegraded":"2023-10-21T05:15:22.312073+0000","last_fullsized":"2023-10-21T05:15:22.312073+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6698,"log_dups_size":0,"ondisk_log_size":6698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"91'484835","reported_seq":964997,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:27.264509+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:15:27.264509+0000","last_peered":"2023-10-21T05:15:27.264509+0000","last_clean":"2023-10-21T05:15:27.264509+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:15:27.264509+0000","last_undegraded":"2023-10-21T05:15:27.264509+0000","last_fullsized":"2023-10-21T05:15:27.264509+0000","mapping_epoch":16,"log_start":"89'474817","ondisk_log_start":"89'474817","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1859,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 467s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6871498752,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":481387,"num_read_kb":1895068,"num_write":482028,"num_write_kb":1928112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":219,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:57.047344+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:14:57.047344+0000","last_peered":"2023-10-21T05:14:57.047344+0000","last_clean":"2023-10-21T05:14:57.047344+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:14:57.047344+0000","last_undegraded":"2023-10-21T05:14:57.047344+0000","last_fullsized":"2023-10-21T05:14:57.047344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"91'488531","reported_seq":974378,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:23.613565+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:15:23.613565+0000","last_peered":"2023-10-21T05:15:23.613565+0000","last_clean":"2023-10-21T05:15:23.613565+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:15:23.613565+0000","last_undegraded":"2023-10-21T05:15:23.613565+0000","last_fullsized":"2023-10-21T05:15:23.613565+0000","mapping_epoch":16,"log_start":"89'478518","ondisk_log_start":"89'478518","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1784,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 452s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6985682944,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":487151,"num_read_kb":1917480,"num_write":485677,"num_write_kb":1942708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3857,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:22.312215+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:15:22.312215+0000","last_peered":"2023-10-21T05:15:22.312215+0000","last_clean":"2023-10-21T05:15:22.312215+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:15:22.312215+0000","last_undegraded":"2023-10-21T05:15:22.312215+0000","last_fullsized":"2023-10-21T05:15:22.312215+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"91'457883","reported_seq":913709,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:27.252947+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:15:27.252947+0000","last_peered":"2023-10-21T05:15:27.252947+0000","last_clean":"2023-10-21T05:15:27.252947+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:15:27.252947+0000","last_undegraded":"2023-10-21T05:15:27.252947+0000","last_fullsized":"2023-10-21T05:15:27.252947+0000","mapping_epoch":16,"log_start":"89'447820","ondisk_log_start":"89'447820","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2472,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 471s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6472867840,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":456879,"num_read_kb":1798192,"num_write":454834,"num_write_kb":1819336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3792,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:22.405675+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:15:22.405675+0000","last_peered":"2023-10-21T05:15:22.405675+0000","last_clean":"2023-10-21T05:15:22.405675+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:15:22.405675+0000","last_undegraded":"2023-10-21T05:15:22.405675+0000","last_fullsized":"2023-10-21T05:15:22.405675+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"91'475639","reported_seq":948229,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:23.611239+0000","last_change":"2023-10-21T05:15:18.965041+0000","last_active":"2023-10-21T05:15:23.611239+0000","last_peered":"2023-10-21T05:15:23.611239+0000","last_clean":"2023-10-21T05:15:23.611239+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:15:23.611239+0000","last_undegraded":"2023-10-21T05:15:23.611239+0000","last_fullsized":"2023-10-21T05:15:23.611239+0000","mapping_epoch":16,"log_start":"89'465624","ondisk_log_start":"89'465624","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6794432531,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":473909,"num_read_kb":1864932,"num_write":473948,"num_write_kb":1895786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"91'16512","reported_seq":22790,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:22.312765+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:15:22.312765+0000","last_peered":"2023-10-21T05:15:22.312765+0000","last_clean":"2023-10-21T05:15:22.312765+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:15:22.312765+0000","last_undegraded":"2023-10-21T05:15:22.312765+0000","last_fullsized":"2023-10-21T05:15:22.312765+0000","mapping_epoch":13,"log_start":"80'6500","ondisk_log_start":"80'6500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":264,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:04.889065+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:14:04.889065+0000","last_peered":"2023-10-21T05:14:04.889065+0000","last_clean":"2023-10-21T05:14:04.889065+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:14:04.889065+0000","last_undegraded":"2023-10-21T05:14:04.889065+0000","last_fullsized":"2023-10-21T05:14:04.889065+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27124482067,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1899326,"num_read_kb":7475672,"num_write":1896487,"num_write_kb":7585942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15532851200,"data_stored":18426822694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40109,"ondisk_log_size":40109,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37193,"ondisk_log_size":37193,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739258,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9129920,"kb_used_data":7582328,"kb_used_omap":786,"kb_used_meta":1546797,"kb_avail":85241920,"statfs":{"total":96636764160,"available":87287726080,"internally_reserved":0,"allocated":7764303872,"data_stored":9210769690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":1583920891},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.56499999999999995,"15min":0.46700000000000003},"min":{"1min":0.184,"5min":0.17899999999999999,"15min":0.153},"max":{"1min":0.52000000000000002,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.55500000000000005,"15min":0.45900000000000002},"min":{"1min":0.252,"5min":0.192,"15min":0.185},"max":{"1min":0.51200000000000001,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.29299999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:14:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.55900000000000005,"15min":0.621},"min":{"1min":0.224,"5min":0.188,"15min":0.188},"max":{"1min":1.238,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.52200000000000002,"15min":0.51200000000000001},"min":{"1min":0.26800000000000002,"5min":0.23200000000000001,"15min":0.20999999999999999},"max":{"1min":0.63300000000000001,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.32900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739255,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5668192,"kb_used_data":3839064,"kb_used_omap":353,"kb_used_meta":1828766,"kb_avail":88703648,"statfs":{"total":96636764160,"available":90832535552,"internally_reserved":0,"allocated":3931201536,"data_stored":4677097836,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1872656599},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,3,6,7,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:53 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.503,"15min":0.51700000000000002},"min":{"1min":0.26400000000000001,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":0.91900000000000004,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":0.61099999999999999,"5min":4.6109999999999998,"15min":4.6109999999999998},"last":0.30399999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:14:53 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.45500000000000002,"15min":0.44400000000000001},"min":{"1min":0.221,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.95499999999999996,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.38100000000000001,"15min":0.39400000000000002},"min":{"1min":0.247,"5min":0.221,"15min":0.20499999999999999},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":4.306},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739259,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5535472,"kb_used_data":3747752,"kb_used_omap":329,"kb_used_meta":1787382,"kb_avail":88836368,"statfs":{"total":96636764160,"available":90968440832,"internally_reserved":0,"allocated":3837698048,"data_stored":4538582333,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1830279178},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,25,0,2,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:14:58 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.36899999999999999,"15min":0.375},"min":{"1min":0.22500000000000001,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":0.505,"5min":1.7430000000000001,"15min":3.7370000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42699999999999999,"15min":0.42999999999999999},"min":{"1min":0.312,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":0.54800000000000004,"5min":1.778,"15min":3.7709999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:14:58 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.55800000000000005,"15min":0.46000000000000002},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.214},"max":{"1min":0.57699999999999996,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61199999999999999,"15min":0.497},"min":{"1min":0.252,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":3.7799999999999998,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3837399040,"data_stored":4538335232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3930624000,"data_stored":4676673555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7764828160,"data_stored":9211813907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:15:29.713 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:29.713 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:29.714 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:29.714 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:29.715 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:15:34.598 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:34.594+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:15:34.605 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:15:34.598+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:15:49.715 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:15:50.385 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:15:50.387 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:15:50.430 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2320,"stamp":"2023-10-21T05:15:50.175914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27152567425,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1926042,"num_read_kb":7522421,"num_write":1905131,"num_write_kb":7619767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77544,"ondisk_log_size":77544,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20584120,"kb_used_data":15236384,"kb_used_omap":1470,"kb_used_meta":5346241,"kb_avail":262531400,"statfs":{"total":289910292480,"available":268832153600,"internally_reserved":0,"allocated":15602057216,"data_stored":18495303619,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1505316,"internal_metadata":5474551772},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,8,5,12,12,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15147008,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4252,"num_read_kb":16960,"num_write":4338,"num_write_kb":17352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002193"},"pg_stats":[{"pgid":"2.7","version":"91'13264","reported_seq":18232,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:47.912715+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:15:47.912715+0000","last_peered":"2023-10-21T05:15:47.912715+0000","last_clean":"2023-10-21T05:15:47.912715+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:15:47.912715+0000","last_undegraded":"2023-10-21T05:15:47.912715+0000","last_fullsized":"2023-10-21T05:15:47.912715+0000","mapping_epoch":13,"log_start":"74'3200","ondisk_log_start":"74'3200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"91'6767","reported_seq":10291,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:47.911982+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:15:47.911982+0000","last_peered":"2023-10-21T05:15:47.911982+0000","last_clean":"2023-10-21T05:15:47.911982+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:15:47.911982+0000","last_undegraded":"2023-10-21T05:15:47.911982+0000","last_fullsized":"2023-10-21T05:15:47.911982+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6767,"log_dups_size":0,"ondisk_log_size":6767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"91'3391","reported_seq":4793,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:47.738487+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:15:47.738487+0000","last_peered":"2023-10-21T05:15:47.738487+0000","last_clean":"2023-10-21T05:15:47.738487+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:15:47.738487+0000","last_undegraded":"2023-10-21T05:15:47.738487+0000","last_fullsized":"2023-10-21T05:15:47.738487+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"91'6715","reported_seq":9648,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:47.912041+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:15:47.912041+0000","last_peered":"2023-10-21T05:15:47.912041+0000","last_clean":"2023-10-21T05:15:47.912041+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:15:47.912041+0000","last_undegraded":"2023-10-21T05:15:47.912041+0000","last_fullsized":"2023-10-21T05:15:47.912041+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6715,"log_dups_size":0,"ondisk_log_size":6715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"91'486685","reported_seq":968621,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:47.399827+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:15:47.399827+0000","last_peered":"2023-10-21T05:15:47.399827+0000","last_clean":"2023-10-21T05:15:47.399827+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:15:47.399827+0000","last_undegraded":"2023-10-21T05:15:47.399827+0000","last_fullsized":"2023-10-21T05:15:47.399827+0000","mapping_epoch":16,"log_start":"89'476617","ondisk_log_start":"89'476617","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1973,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 487s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6877712384,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":483149,"num_read_kb":1902104,"num_write":483812,"num_write_kb":1935248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":219,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:57.047344+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:14:57.047344+0000","last_peered":"2023-10-21T05:14:57.047344+0000","last_clean":"2023-10-21T05:14:57.047344+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:14:57.047344+0000","last_undegraded":"2023-10-21T05:14:57.047344+0000","last_fullsized":"2023-10-21T05:14:57.047344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"91'490875","reported_seq":978907,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:48.703416+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:15:48.703416+0000","last_peered":"2023-10-21T05:15:48.703416+0000","last_clean":"2023-10-21T05:15:48.703416+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:15:48.703416+0000","last_undegraded":"2023-10-21T05:15:48.703416+0000","last_fullsized":"2023-10-21T05:15:48.703416+0000","mapping_epoch":16,"log_start":"89'480818","ondisk_log_start":"89'480818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1908,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 478s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":6993694720,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":489322,"num_read_kb":1926128,"num_write":487940,"num_write_kb":1951760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3872,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:47.954355+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:15:47.954355+0000","last_peered":"2023-10-21T05:15:47.954355+0000","last_clean":"2023-10-21T05:15:47.954355+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:15:47.954355+0000","last_undegraded":"2023-10-21T05:15:47.954355+0000","last_fullsized":"2023-10-21T05:15:47.954355+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"91'459623","reported_seq":917055,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:15:47.399981+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:15:47.399981+0000","last_peered":"2023-10-21T05:15:47.399981+0000","last_clean":"2023-10-21T05:15:47.399981+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:15:47.399981+0000","last_undegraded":"2023-10-21T05:15:47.399981+0000","last_fullsized":"2023-10-21T05:15:47.399981+0000","mapping_epoch":16,"log_start":"89'449620","ondisk_log_start":"89'449620","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2595,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 491s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6478516224,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":458472,"num_read_kb":1804540,"num_write":456498,"num_write_kb":1825992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3807,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:47.913195+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:15:47.913195+0000","last_peered":"2023-10-21T05:15:47.913195+0000","last_clean":"2023-10-21T05:15:47.913195+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:15:47.913195+0000","last_undegraded":"2023-10-21T05:15:47.913195+0000","last_fullsized":"2023-10-21T05:15:47.913195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"91'477834","reported_seq":952526,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:48.758662+0000","last_change":"2023-10-21T05:15:43.227441+0000","last_active":"2023-10-21T05:15:48.758662+0000","last_peered":"2023-10-21T05:15:48.758662+0000","last_clean":"2023-10-21T05:15:48.758662+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:15:48.758662+0000","last_undegraded":"2023-10-21T05:15:48.758662+0000","last_fullsized":"2023-10-21T05:15:48.758662+0000","mapping_epoch":16,"log_start":"89'467824","ondisk_log_start":"89'467824","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6802051091,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":476011,"num_read_kb":1873328,"num_write":476143,"num_write_kb":1904566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"91'16552","reported_seq":22855,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:47.912753+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:15:47.912753+0000","last_peered":"2023-10-21T05:15:47.912753+0000","last_clean":"2023-10-21T05:15:47.912753+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:15:47.912753+0000","last_undegraded":"2023-10-21T05:15:47.912753+0000","last_fullsized":"2023-10-21T05:15:47.912753+0000","mapping_epoch":13,"log_start":"80'6500","ondisk_log_start":"80'6500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":264,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:04.889065+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:14:04.889065+0000","last_peered":"2023-10-21T05:14:04.889065+0000","last_clean":"2023-10-21T05:14:04.889065+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:14:04.889065+0000","last_undegraded":"2023-10-21T05:14:04.889065+0000","last_fullsized":"2023-10-21T05:14:04.889065+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27151974419,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1906954,"num_read_kb":7506100,"num_write":1904393,"num_write_kb":7617566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15601356800,"data_stored":18495328294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40138,"ondisk_log_size":40138,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37307,"ondisk_log_size":37307,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739263,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9255700,"kb_used_data":7617868,"kb_used_omap":786,"kb_used_meta":1637037,"kb_avail":85116140,"statfs":{"total":96636764160,"available":87158927360,"internally_reserved":0,"allocated":7800696832,"data_stored":9247162650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":1676326651},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.56499999999999995,"15min":0.46700000000000003},"min":{"1min":0.184,"5min":0.17899999999999999,"15min":0.153},"max":{"1min":0.52000000000000002,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.55500000000000005,"15min":0.45900000000000002},"min":{"1min":0.252,"5min":0.192,"15min":0.185},"max":{"1min":0.51200000000000001,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.57399999999999995}]},{"osd":1,"last update":"Sat Oct 21 05:14:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.55900000000000005,"15min":0.621},"min":{"1min":0.224,"5min":0.188,"15min":0.188},"max":{"1min":1.238,"5min":7.8449999999999998,"15min":9.2569999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.52200000000000002,"15min":0.51200000000000001},"min":{"1min":0.26800000000000002,"5min":0.23200000000000001,"15min":0.20999999999999999},"max":{"1min":0.63300000000000001,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.35499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739260,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5747204,"kb_used_data":3856956,"kb_used_omap":353,"kb_used_meta":1889886,"kb_avail":88624636,"statfs":{"total":96636764160,"available":90751627264,"internally_reserved":0,"allocated":3949522944,"data_stored":4695419244,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":1935243479},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,0,2,6,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:14:53 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.503,"15min":0.51700000000000002},"min":{"1min":0.26400000000000001,"5min":0.17000000000000001,"15min":0.16900000000000001},"max":{"1min":0.91900000000000004,"5min":4.5599999999999996,"15min":4.5599999999999996},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":0.61099999999999999,"5min":4.6109999999999998,"15min":4.6109999999999998},"last":0.33300000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:14:53 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.45500000000000002,"15min":0.44400000000000001},"min":{"1min":0.221,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.95499999999999996,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.38100000000000001,"15min":0.39400000000000002},"min":{"1min":0.247,"5min":0.221,"15min":0.20499999999999999},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":4.306},"last":0.35999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739263,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5581216,"kb_used_data":3761560,"kb_used_omap":329,"kb_used_meta":1819318,"kb_avail":88790624,"statfs":{"total":96636764160,"available":90921598976,"internally_reserved":0,"allocated":3851837440,"data_stored":4552721725,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1862981642},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2,2,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:14:58 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.36899999999999999,"15min":0.375},"min":{"1min":0.22500000000000001,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":0.505,"5min":1.7430000000000001,"15min":3.7370000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42699999999999999,"15min":0.42999999999999999},"min":{"1min":0.312,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":0.54800000000000004,"5min":1.778,"15min":3.7709999999999999},"last":0.35999999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:14:58 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.55800000000000005,"15min":0.46000000000000002},"min":{"1min":0.25900000000000001,"5min":0.22,"15min":0.214},"max":{"1min":0.57699999999999996,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61199999999999999,"15min":0.497},"min":{"1min":0.252,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":3.7799999999999998,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.39200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3851513856,"data_stored":4552450048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3948904448,"data_stored":4694954003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7800938496,"data_stored":9247924243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:15:50.432 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:50.432 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:50.433 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:50.433 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:15:50.434 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:16:10.436 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:16:11.051 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:16:11.051 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:16:11.088 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2330,"stamp":"2023-10-21T05:16:10.179468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27177151617,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1932708,"num_read_kb":7549021,"num_write":1912210,"num_write_kb":7648083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77516,"ondisk_log_size":77516,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20872256,"kb_used_data":15292776,"kb_used_omap":1470,"kb_used_meta":5577985,"kb_avail":262243264,"statfs":{"total":289910292480,"available":268537102336,"internally_reserved":0,"allocated":15659802624,"data_stored":18553049027,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1505316,"internal_metadata":5711857628},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,5,14,16,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15405056,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4168,"num_read_kb":16636,"num_write":4430,"num_write_kb":17720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002026"},"pg_stats":[{"pgid":"2.7","version":"91'13304","reported_seq":18288,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:16:08.392727+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:16:08.392727+0000","last_peered":"2023-10-21T05:16:08.392727+0000","last_clean":"2023-10-21T05:16:08.392727+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:16:08.392727+0000","last_undegraded":"2023-10-21T05:16:08.392727+0000","last_fullsized":"2023-10-21T05:16:08.392727+0000","mapping_epoch":13,"log_start":"74'3300","ondisk_log_start":"74'3300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"91'6787","reported_seq":10319,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:16:08.392115+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:16:08.392115+0000","last_peered":"2023-10-21T05:16:08.392115+0000","last_clean":"2023-10-21T05:16:08.392115+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:16:08.392115+0000","last_undegraded":"2023-10-21T05:16:08.392115+0000","last_fullsized":"2023-10-21T05:16:08.392115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6787,"log_dups_size":0,"ondisk_log_size":6787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"91'3401","reported_seq":4807,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:16:08.113388+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:16:08.113388+0000","last_peered":"2023-10-21T05:16:08.113388+0000","last_clean":"2023-10-21T05:16:08.113388+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:16:08.113388+0000","last_undegraded":"2023-10-21T05:16:08.113388+0000","last_fullsized":"2023-10-21T05:16:08.113388+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3401,"log_dups_size":0,"ondisk_log_size":3401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"91'6735","reported_seq":9676,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:16:08.392492+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:16:08.392492+0000","last_peered":"2023-10-21T05:16:08.392492+0000","last_clean":"2023-10-21T05:16:08.392492+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:16:08.392492+0000","last_undegraded":"2023-10-21T05:16:08.392492+0000","last_fullsized":"2023-10-21T05:16:08.392492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6735,"log_dups_size":0,"ondisk_log_size":6735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"91'488469","reported_seq":972111,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:07.466552+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:16:07.466552+0000","last_peered":"2023-10-21T05:16:07.466552+0000","last_clean":"2023-10-21T05:16:07.466552+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:16:07.466552+0000","last_undegraded":"2023-10-21T05:16:07.466552+0000","last_fullsized":"2023-10-21T05:16:07.466552+0000","mapping_epoch":16,"log_start":"90'478417","ondisk_log_start":"90'478417","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2071,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 507s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6883704832,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":484842,"num_read_kb":1908864,"num_write":485540,"num_write_kb":1942160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":219,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-21T05:14:57.047344+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:14:57.047344+0000","last_peered":"2023-10-21T05:14:57.047344+0000","last_clean":"2023-10-21T05:14:57.047344+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:14:57.047344+0000","last_undegraded":"2023-10-21T05:14:57.047344+0000","last_fullsized":"2023-10-21T05:14:57.047344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"91'492729","reported_seq":982488,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:08.852271+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:16:08.852271+0000","last_peered":"2023-10-21T05:16:08.852271+0000","last_clean":"2023-10-21T05:16:08.852271+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:16:08.852271+0000","last_undegraded":"2023-10-21T05:16:08.852271+0000","last_fullsized":"2023-10-21T05:16:08.852271+0000","mapping_epoch":16,"log_start":"90'482718","ondisk_log_start":"90'482718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1972,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 498s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7000158208,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":491041,"num_read_kb":1932992,"num_write":489754,"num_write_kb":1959016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3884,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:16:08.417179+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:16:08.417179+0000","last_peered":"2023-10-21T05:16:08.417179+0000","last_clean":"2023-10-21T05:16:08.417179+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:16:08.417179+0000","last_undegraded":"2023-10-21T05:16:08.417179+0000","last_fullsized":"2023-10-21T05:16:08.417179+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"91'461407","reported_seq":920396,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:07.452233+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:16:07.452233+0000","last_peered":"2023-10-21T05:16:07.452233+0000","last_clean":"2023-10-21T05:16:07.452233+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:16:07.452233+0000","last_undegraded":"2023-10-21T05:16:07.452233+0000","last_fullsized":"2023-10-21T05:16:07.452233+0000","mapping_epoch":16,"log_start":"90'451320","ondisk_log_start":"90'451320","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2687,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 511s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6484332544,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":460023,"num_read_kb":1810728,"num_write":458225,"num_write_kb":1832900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3820,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:16:08.394189+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:16:08.394189+0000","last_peered":"2023-10-21T05:16:08.394189+0000","last_clean":"2023-10-21T05:16:08.394189+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:16:08.394189+0000","last_undegraded":"2023-10-21T05:16:08.394189+0000","last_fullsized":"2023-10-21T05:16:08.394189+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"91'479644","reported_seq":956049,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:16:08.843464+0000","last_change":"2023-10-21T05:16:04.288268+0000","last_active":"2023-10-21T05:16:08.843464+0000","last_peered":"2023-10-21T05:16:08.843464+0000","last_clean":"2023-10-21T05:16:08.843464+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:16:08.843464+0000","last_undegraded":"2023-10-21T05:16:08.843464+0000","last_fullsized":"2023-10-21T05:16:08.843464+0000","mapping_epoch":16,"log_start":"90'469624","ondisk_log_start":"90'469624","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6808363027,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":477714,"num_read_kb":1880116,"num_write":477953,"num_write_kb":1911806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"91'16602","reported_seq":22925,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:16:08.393263+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:16:08.393263+0000","last_peered":"2023-10-21T05:16:08.393263+0000","last_clean":"2023-10-21T05:16:08.393263+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:16:08.393263+0000","last_undegraded":"2023-10-21T05:16:08.393263+0000","last_fullsized":"2023-10-21T05:16:08.393263+0000","mapping_epoch":13,"log_start":"80'6600","ondisk_log_start":"80'6600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":265,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:14.916866+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:15:14.916866+0000","last_peered":"2023-10-21T05:15:14.916866+0000","last_clean":"2023-10-21T05:15:14.916866+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:15:14.916866+0000","last_undegraded":"2023-10-21T05:15:14.916866+0000","last_fullsized":"2023-10-21T05:15:14.916866+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27176558611,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1913620,"num_read_kb":7532700,"num_write":1911472,"num_write_kb":7645882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15659536384,"data_stored":18553507878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40170,"ondisk_log_size":40170,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37247,"ondisk_log_size":37247,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739267,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9390704,"kb_used_data":7646056,"kb_used_omap":786,"kb_used_meta":1743853,"kb_avail":84981136,"statfs":{"total":96636764160,"available":87020683264,"internally_reserved":0,"allocated":7829561344,"data_stored":9276027162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":1785706235},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":0.48199999999999998,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.19},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.496,"15min":0.46000000000000002},"min":{"1min":0.246,"5min":0.192,"15min":0.185},"max":{"1min":0.57399999999999995,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.26000000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.56499999999999995,"15min":0.60199999999999998},"min":{"1min":0.219,"5min":0.188,"15min":0.188},"max":{"1min":5.2380000000000004,"5min":5.2380000000000004,"15min":9.2569999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52800000000000002,"15min":0.50800000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.8300000000000001,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.23699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739264,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5833864,"kb_used_data":3871424,"kb_used_omap":353,"kb_used_meta":1962078,"kb_avail":88537976,"statfs":{"total":96636764160,"available":90662887424,"internally_reserved":0,"allocated":3964338176,"data_stored":4710234476,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":362281,"internal_metadata":2009168087},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,6,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.432,"15min":0.49399999999999999},"min":{"1min":0.19600000000000001,"5min":0.19,"15min":0.16900000000000001},"max":{"1min":0.48999999999999999,"5min":3.633,"15min":4.5599999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41299999999999998,"15min":0.46200000000000002},"min":{"1min":0.246,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":0.52500000000000002,"5min":3.5870000000000002,"15min":4.6109999999999998},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.48999999999999999,"15min":0.45700000000000002},"min":{"1min":0.221,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.3259999999999996,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.436},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.28399999999999997,"5min":0.221,"15min":0.20499999999999999},"max":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.306},"last":0.38600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739267,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5647688,"kb_used_data":3775296,"kb_used_omap":329,"kb_used_meta":1872054,"kb_avail":88724152,"statfs":{"total":96636764160,"available":90853531648,"internally_reserved":0,"allocated":3865903104,"data_stored":4566787389,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1916983306},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,6,2,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:16:03 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.42399999999999999,"15min":0.38500000000000001},"min":{"1min":0.20000000000000001,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":5.0099999999999998,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":0.317},{"interface":"front","average":{"1min":0.754,"5min":0.48199999999999998,"15min":0.439},"min":{"1min":0.29199999999999998,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":5.0469999999999997,"5min":5.0469999999999997,"15min":5.0469999999999997},"last":0.35399999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:16:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.57899999999999996,"15min":0.47399999999999998},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":4.2789999999999999,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.59399999999999997,"15min":0.52000000000000002},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":4.1340000000000003,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3865665536,"data_stored":4566601728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3963731968,"data_stored":4709781523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7830138880,"data_stored":9277124627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:16:11.090 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:11.091 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:11.091 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:11.092 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:11.092 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:16:19.198 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:19.186+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:16:19.200 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:19.190+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:16:19.200 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:19.190+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:16:19.200 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:19.190+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:16:19.200 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:19.190+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:16:19.206 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:19.198+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:16:19.206 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:19.198+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:16:19.208 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:19.202+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:16:19.209 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:19.202+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:16:19.210 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:19.206+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:16:19.210 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:19.206+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:16:19.211 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:19.206+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:16:31.094 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:16:31.325 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.318+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:16:31.340 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.334+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:16:31.341 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.334+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:16:31.341 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.334+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:16:31.342 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.334+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T05:16:31.342 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.334+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:16:31.344 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.338+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:16:31.344 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.338+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:16:31.347 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.342+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:16:31.347 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.342+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:16:31.355 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.350+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:16:31.355 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.350+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:16:31.355 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:31.350+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:16:31.656 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:16:31.657 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:16:31.694 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2341,"stamp":"2023-10-21T05:16:30.182879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27201952897,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1939662,"num_read_kb":7576797,"num_write":1919354,"num_write_kb":7676659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77733,"ondisk_log_size":77733,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20472152,"kb_used_data":15351808,"kb_used_omap":1471,"kb_used_meta":5118848,"kb_avail":262643368,"statfs":{"total":289910292480,"available":268946808832,"internally_reserved":0,"allocated":15720251392,"data_stored":18613484334,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1506995,"internal_metadata":5241700685},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,2,10,12,12,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12255232,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3596,"num_read_kb":14364,"num_write":3543,"num_write_kb":14172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001692"},"pg_stats":[{"pgid":"2.7","version":"92'13376","reported_seq":18377,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:28.928724+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:16:28.928724+0000","last_peered":"2023-10-21T05:16:28.928724+0000","last_clean":"2023-10-21T05:16:28.928724+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:16:28.928724+0000","last_undegraded":"2023-10-21T05:16:28.928724+0000","last_fullsized":"2023-10-21T05:16:28.928724+0000","mapping_epoch":13,"log_start":"74'3300","ondisk_log_start":"74'3300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'6823","reported_seq":10364,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:28.911363+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:16:28.911363+0000","last_peered":"2023-10-21T05:16:28.911363+0000","last_clean":"2023-10-21T05:16:28.911363+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:16:28.911363+0000","last_undegraded":"2023-10-21T05:16:28.911363+0000","last_fullsized":"2023-10-21T05:16:28.911363+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6823,"log_dups_size":0,"ondisk_log_size":6823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"92'3419","reported_seq":4830,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:28.872892+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:16:28.872892+0000","last_peered":"2023-10-21T05:16:28.872892+0000","last_clean":"2023-10-21T05:16:28.872892+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:16:28.872892+0000","last_undegraded":"2023-10-21T05:16:28.872892+0000","last_fullsized":"2023-10-21T05:16:28.872892+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3419,"log_dups_size":0,"ondisk_log_size":3419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"92'6771","reported_seq":9721,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:28.911293+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:16:28.911293+0000","last_peered":"2023-10-21T05:16:28.911293+0000","last_clean":"2023-10-21T05:16:28.911293+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:16:28.911293+0000","last_undegraded":"2023-10-21T05:16:28.911293+0000","last_fullsized":"2023-10-21T05:16:28.911293+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6771,"log_dups_size":0,"ondisk_log_size":6771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"92'490307","reported_seq":975718,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:27.723266+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:16:27.723266+0000","last_peered":"2023-10-21T05:16:27.723266+0000","last_clean":"2023-10-21T05:16:27.723266+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:16:27.723266+0000","last_undegraded":"2023-10-21T05:16:27.723266+0000","last_fullsized":"2023-10-21T05:16:27.723266+0000","mapping_epoch":16,"log_start":"90'480217","ondisk_log_start":"90'480217","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2148,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 527s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6889926656,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":486603,"num_read_kb":1915904,"num_write":487331,"num_write_kb":1949324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":220,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:14.882672+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:15:14.882672+0000","last_peered":"2023-10-21T05:15:14.882672+0000","last_clean":"2023-10-21T05:15:14.882672+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:15:14.882672+0000","last_undegraded":"2023-10-21T05:15:14.882672+0000","last_fullsized":"2023-10-21T05:15:14.882672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"92'494533","reported_seq":986115,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:28.991327+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:16:28.991327+0000","last_peered":"2023-10-21T05:16:28.991327+0000","last_clean":"2023-10-21T05:16:28.991327+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:16:28.991327+0000","last_undegraded":"2023-10-21T05:16:28.991327+0000","last_fullsized":"2023-10-21T05:16:28.991327+0000","mapping_epoch":16,"log_start":"90'484518","ondisk_log_start":"90'484518","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2024,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 518s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7006326784,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":492853,"num_read_kb":1940236,"num_write":491527,"num_write_kb":1966108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3894,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:16:23.758443+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:16:23.758443+0000","last_peered":"2023-10-21T05:16:23.758443+0000","last_clean":"2023-10-21T05:16:23.758443+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:16:23.758443+0000","last_undegraded":"2023-10-21T05:16:23.758443+0000","last_fullsized":"2023-10-21T05:16:23.758443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"92'463248","reported_seq":923942,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:27.720063+0000","last_change":"2023-10-21T05:07:35.822355+0000","last_active":"2023-10-21T05:16:27.720063+0000","last_peered":"2023-10-21T05:16:27.720063+0000","last_clean":"2023-10-21T05:16:27.720063+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:16:27.720063+0000","last_undegraded":"2023-10-21T05:16:27.720063+0000","last_fullsized":"2023-10-21T05:16:27.720063+0000","mapping_epoch":16,"log_start":"90'453220","ondisk_log_start":"90'453220","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2753,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 531s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6490472448,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":461710,"num_read_kb":1817464,"num_write":460023,"num_write_kb":1840092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3830,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:16:23.753947+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:16:23.753947+0000","last_peered":"2023-10-21T05:16:23.753947+0000","last_clean":"2023-10-21T05:16:23.753947+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:16:23.753947+0000","last_undegraded":"2023-10-21T05:16:23.753947+0000","last_fullsized":"2023-10-21T05:16:23.753947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"92'481426","reported_seq":959530,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:28.988919+0000","last_change":"2023-10-21T05:16:27.459419+0000","last_active":"2023-10-21T05:16:28.988919+0000","last_peered":"2023-10-21T05:16:28.988919+0000","last_clean":"2023-10-21T05:16:28.988919+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:16:28.988919+0000","last_undegraded":"2023-10-21T05:16:28.988919+0000","last_fullsized":"2023-10-21T05:16:28.988919+0000","mapping_epoch":16,"log_start":"90'471424","ondisk_log_start":"90'471424","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6814634003,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":479408,"num_read_kb":1886872,"num_write":479735,"num_write_kb":1918934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'16692","reported_seq":23036,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:28.987479+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:16:28.987479+0000","last_peered":"2023-10-21T05:16:28.987479+0000","last_clean":"2023-10-21T05:16:28.987479+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:16:28.987479+0000","last_undegraded":"2023-10-21T05:16:28.987479+0000","last_fullsized":"2023-10-21T05:16:28.987479+0000","mapping_epoch":13,"log_start":"80'6600","ondisk_log_start":"80'6600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":266,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:58.808976+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:15:58.808976+0000","last_peered":"2023-10-21T05:15:58.808976+0000","last_clean":"2023-10-21T05:15:58.808976+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:15:58.808976+0000","last_undegraded":"2023-10-21T05:15:58.808976+0000","last_fullsized":"2023-10-21T05:15:58.808976+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27201359891,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1920574,"num_read_kb":7560476,"num_write":1918616,"num_write_kb":7674458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15717928960,"data_stored":18611900454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40135,"ondisk_log_size":40135,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37499,"ondisk_log_size":37499,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739271,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9506636,"kb_used_data":7675908,"kb_used_omap":786,"kb_used_meta":1829933,"kb_avail":84865204,"statfs":{"total":96636764160,"available":86901968896,"internally_reserved":0,"allocated":7860129792,"data_stored":9306591123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":1873852155},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":0.48199999999999998,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.496,"15min":0.46000000000000002},"min":{"1min":0.246,"5min":0.192,"15min":0.185},"max":{"1min":0.57399999999999995,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.621}]},{"osd":1,"last update":"Sat Oct 21 05:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.56499999999999995,"15min":0.60199999999999998},"min":{"1min":0.219,"5min":0.188,"15min":0.188},"max":{"1min":5.2380000000000004,"5min":5.2380000000000004,"15min":9.2569999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52800000000000002,"15min":0.50800000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.8300000000000001,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.53300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739268,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5270896,"kb_used_data":3885608,"kb_used_omap":355,"kb_used_meta":1384924,"kb_avail":89100944,"statfs":{"total":96636764160,"available":91239366656,"internally_reserved":0,"allocated":3978862592,"data_stored":4724754405,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":1418162760},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,4,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.432,"15min":0.49399999999999999},"min":{"1min":0.19600000000000001,"5min":0.19,"15min":0.16900000000000001},"max":{"1min":0.48999999999999999,"5min":3.633,"15min":4.5599999999999996},"last":1.129},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41299999999999998,"15min":0.46200000000000002},"min":{"1min":0.246,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":0.52500000000000002,"5min":3.5870000000000002,"15min":4.6109999999999998},"last":1.153}]},{"osd":2,"last update":"Sat Oct 21 05:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.48999999999999999,"15min":0.45700000000000002},"min":{"1min":0.221,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.3259999999999996,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.217},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.28399999999999997,"5min":0.221,"15min":0.20499999999999999},"max":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.306},"last":0.28199999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739272,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5694620,"kb_used_data":3790292,"kb_used_omap":329,"kb_used_meta":1903990,"kb_avail":88677220,"statfs":{"total":96636764160,"available":90805473280,"internally_reserved":0,"allocated":3881259008,"data_stored":4582138806,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":1949685770},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2,8,5,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:16:03 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.42399999999999999,"15min":0.38500000000000001},"min":{"1min":0.20000000000000001,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":5.0099999999999998,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":0.314},{"interface":"front","average":{"1min":0.754,"5min":0.48199999999999998,"15min":0.439},"min":{"1min":0.29199999999999998,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":5.0469999999999997,"5min":5.0469999999999997,"15min":5.0469999999999997},"last":0.33800000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:16:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.57899999999999996,"15min":0.47399999999999998},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":4.2789999999999999,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.59399999999999997,"15min":0.52000000000000002},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":4.1340000000000003,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3880366080,"data_stored":4581302272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3978297344,"data_stored":4724346899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7859265536,"data_stored":9306251283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:16:31.696 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:31.696 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:31.697 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:31.698 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:31.698 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:16:32.317 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-21T05:16:32.348 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:16:32.895 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:16:32.928 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-21T05:16:33.471 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-21T05:16:33.535 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:16:34.089 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:16:34.133 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:16:35.603 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:35.594+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:16:35.618 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:35.598+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:16:35.618 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:35.602+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:16:35.618 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:35.602+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:16:44.893 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:44.886+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:16:44.894 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:44.886+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:16:44.894 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:44.886+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:16:44.894 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:44.886+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:16:44.895 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:44.886+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:16:44.903 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:44.898+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:16:44.917 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:44.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:16:44.920 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:44.914+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:16:44.921 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:44.914+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:16:52.428 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:52.418+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:16:52.429 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:52.418+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:16:52.429 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:52.418+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:16:52.440 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:52.430+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:16:52.448 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:52.442+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:16:52.449 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:16:52.442+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:16:54.134 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:16:54.652 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:16:54.652 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:16:54.680 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2353,"stamp":"2023-10-21T05:16:54.187026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27228793985,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1947540,"num_read_kb":7608229,"num_write":1927115,"num_write_kb":7707703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77752,"ondisk_log_size":77752,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20762616,"kb_used_data":15415904,"kb_used_omap":1471,"kb_used_meta":5345216,"kb_avail":262352904,"statfs":{"total":289910292480,"available":268649373696,"internally_reserved":0,"allocated":15785885696,"data_stored":18679118638,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1506995,"internal_metadata":5473501517},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,4,5,10,10,13,19,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15179776,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4322,"num_read_kb":17240,"num_write":4362,"num_write_kb":17448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002004"},"pg_stats":[{"pgid":"2.7","version":"92'13425","reported_seq":18438,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:48.347045+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:16:48.347045+0000","last_peered":"2023-10-21T05:16:48.347045+0000","last_clean":"2023-10-21T05:16:48.347045+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:16:48.347045+0000","last_undegraded":"2023-10-21T05:16:48.347045+0000","last_fullsized":"2023-10-21T05:16:48.347045+0000","mapping_epoch":13,"log_start":"76'3400","ondisk_log_start":"76'3400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'6848","reported_seq":10395,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:48.335945+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:16:48.335945+0000","last_peered":"2023-10-21T05:16:48.335945+0000","last_clean":"2023-10-21T05:16:48.335945+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:16:48.335945+0000","last_undegraded":"2023-10-21T05:16:48.335945+0000","last_fullsized":"2023-10-21T05:16:48.335945+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"92'3432","reported_seq":4846,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:46.342187+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:16:46.342187+0000","last_peered":"2023-10-21T05:16:46.342187+0000","last_clean":"2023-10-21T05:16:46.342187+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:16:46.342187+0000","last_undegraded":"2023-10-21T05:16:46.342187+0000","last_fullsized":"2023-10-21T05:16:46.342187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3432,"log_dups_size":0,"ondisk_log_size":3432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"92'6795","reported_seq":9751,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:48.332388+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:16:48.332388+0000","last_peered":"2023-10-21T05:16:48.332388+0000","last_clean":"2023-10-21T05:16:48.332388+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:16:48.332388+0000","last_undegraded":"2023-10-21T05:16:48.332388+0000","last_fullsized":"2023-10-21T05:16:48.332388+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6795,"log_dups_size":0,"ondisk_log_size":6795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"92'492537","reported_seq":980196,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:52.846607+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:16:52.846607+0000","last_peered":"2023-10-21T05:16:52.846607+0000","last_clean":"2023-10-21T05:16:52.846607+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:16:52.846607+0000","last_undegraded":"2023-10-21T05:16:52.846607+0000","last_fullsized":"2023-10-21T05:16:52.846607+0000","mapping_epoch":16,"log_start":"90'482518","ondisk_log_start":"90'482518","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2248,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 553s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6897352704,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":488833,"num_read_kb":1924808,"num_write":489498,"num_write_kb":1957992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":220,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:14.882672+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:15:14.882672+0000","last_peered":"2023-10-21T05:15:14.882672+0000","last_clean":"2023-10-21T05:15:14.882672+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:15:14.882672+0000","last_undegraded":"2023-10-21T05:15:14.882672+0000","last_fullsized":"2023-10-21T05:15:14.882672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"92'496343","reported_seq":989795,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:49.164316+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:16:49.164316+0000","last_peered":"2023-10-21T05:16:49.164316+0000","last_clean":"2023-10-21T05:16:49.164316+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:16:49.164316+0000","last_undegraded":"2023-10-21T05:16:49.164316+0000","last_fullsized":"2023-10-21T05:16:49.164316+0000","mapping_epoch":16,"log_start":"90'486318","ondisk_log_start":"90'486318","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2134,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 538s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7012532224,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":494711,"num_read_kb":1947640,"num_write":493268,"num_write_kb":1973072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3910,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:49.365900+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:16:49.365900+0000","last_peered":"2023-10-21T05:16:49.365900+0000","last_clean":"2023-10-21T05:16:49.365900+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:16:49.365900+0000","last_undegraded":"2023-10-21T05:16:49.365900+0000","last_fullsized":"2023-10-21T05:16:49.365900+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"92'465281","reported_seq":928056,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:52.837068+0000","last_change":"2023-10-21T05:16:50.922715+0000","last_active":"2023-10-21T05:16:52.837068+0000","last_peered":"2023-10-21T05:16:52.837068+0000","last_clean":"2023-10-21T05:16:52.837068+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:16:52.837068+0000","last_undegraded":"2023-10-21T05:16:52.837068+0000","last_fullsized":"2023-10-21T05:16:52.837068+0000","mapping_epoch":16,"log_start":"90'455220","ondisk_log_start":"90'455220","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2779,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"queued for deep scrub","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6497288192,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":463774,"num_read_kb":1825704,"num_write":462039,"num_write_kb":1848156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3845,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:49.403756+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:16:49.403756+0000","last_peered":"2023-10-21T05:16:49.403756+0000","last_clean":"2023-10-21T05:16:49.403756+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:16:49.403756+0000","last_undegraded":"2023-10-21T05:16:49.403756+0000","last_fullsized":"2023-10-21T05:16:49.403756+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"92'483301","reported_seq":963143,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:16:49.170409+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:16:49.170409+0000","last_peered":"2023-10-21T05:16:49.170409+0000","last_clean":"2023-10-21T05:16:49.170409+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:16:49.170409+0000","last_undegraded":"2023-10-21T05:16:49.170409+0000","last_fullsized":"2023-10-21T05:16:49.170409+0000","mapping_epoch":16,"log_start":"90'473224","ondisk_log_start":"90'473224","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":58,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 15s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6821027859,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":481134,"num_read_kb":1893756,"num_write":481572,"num_write_kb":1926282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'16753","reported_seq":23112,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:48.416541+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:16:48.416541+0000","last_peered":"2023-10-21T05:16:48.416541+0000","last_clean":"2023-10-21T05:16:48.416541+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:16:48.416541+0000","last_undegraded":"2023-10-21T05:16:48.416541+0000","last_fullsized":"2023-10-21T05:16:48.416541+0000","mapping_epoch":13,"log_start":"80'6700","ondisk_log_start":"80'6700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":266,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:58.808976+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:15:58.808976+0000","last_peered":"2023-10-21T05:15:58.808976+0000","last_clean":"2023-10-21T05:15:58.808976+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:15:58.808976+0000","last_undegraded":"2023-10-21T05:15:58.808976+0000","last_fullsized":"2023-10-21T05:15:58.808976+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27228200979,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1928452,"num_read_kb":7591908,"num_write":1926377,"num_write_kb":7705502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15785201664,"data_stored":18679173158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40182,"ondisk_log_size":40182,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37471,"ondisk_log_size":37471,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739276,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9641584,"kb_used_data":7709416,"kb_used_omap":786,"kb_used_meta":1931373,"kb_avail":84730256,"statfs":{"total":96636764160,"available":86763782144,"internally_reserved":0,"allocated":7894441984,"data_stored":9340903315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":1977726715},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":0.48199999999999998,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.496,"15min":0.46000000000000002},"min":{"1min":0.246,"5min":0.192,"15min":0.185},"max":{"1min":0.57399999999999995,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.35699999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.56499999999999995,"15min":0.60199999999999998},"min":{"1min":0.219,"5min":0.188,"15min":0.188},"max":{"1min":5.2380000000000004,"5min":5.2380000000000004,"15min":9.2569999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52800000000000002,"15min":0.50800000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.8300000000000001,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.311}]}]},{"osd":1,"up_from":8,"seq":34359739272,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5351996,"kb_used_data":3900084,"kb_used_omap":355,"kb_used_meta":1451548,"kb_avail":89019844,"statfs":{"total":96636764160,"available":91156320256,"internally_reserved":0,"allocated":3993686016,"data_stored":4739577829,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":1486385736},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,2,3,5,7,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.432,"15min":0.49399999999999999},"min":{"1min":0.19600000000000001,"5min":0.19,"15min":0.16900000000000001},"max":{"1min":0.48999999999999999,"5min":3.633,"15min":4.5599999999999996},"last":0.496},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41299999999999998,"15min":0.46200000000000002},"min":{"1min":0.246,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":0.52500000000000002,"5min":3.5870000000000002,"15min":4.6109999999999998},"last":0.51100000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.48999999999999999,"15min":0.45700000000000002},"min":{"1min":0.221,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.3259999999999996,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.28399999999999997,"5min":0.221,"15min":0.20499999999999999},"max":{"1min":4.1920000000000002,"5min":4.1920000000000002,"15min":4.306},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739277,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5769036,"kb_used_data":3806404,"kb_used_omap":329,"kb_used_meta":1962294,"kb_avail":88602804,"statfs":{"total":96636764160,"available":90729271296,"internally_reserved":0,"allocated":3897757696,"data_stored":4598637494,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":337910,"internal_metadata":2009389066},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,6,2,6,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:16:03 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.42399999999999999,"15min":0.38500000000000001},"min":{"1min":0.20000000000000001,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":5.0099999999999998,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.48199999999999998,"15min":0.439},"min":{"1min":0.29199999999999998,"5min":0.24399999999999999,"15min":0.217},"max":{"1min":5.0469999999999997,"5min":5.0469999999999997,"15min":5.0469999999999997},"last":0.36299999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:16:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.57899999999999996,"15min":0.47399999999999998},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":4.2789999999999999,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.59399999999999997,"15min":0.52000000000000002},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":4.1340000000000003,"5min":6.3710000000000004,"15min":6.3710000000000004},"last":0.46400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3897495552,"data_stored":4598431744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3992944640,"data_stored":4738994195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7894761472,"data_stored":9341747219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:16:54.682 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:54.682 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:54.683 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:16:54.683 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:14.684 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:17:15.325 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:17:15.326 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:17:15.372 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2363,"stamp":"2023-10-21T05:17:14.190479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27253345409,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1954678,"num_read_kb":7636689,"num_write":1934207,"num_write_kb":7736071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77887,"ondisk_log_size":77887,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20374304,"kb_used_data":15472900,"kb_used_omap":1473,"kb_used_meta":4899902,"kb_avail":262741216,"statfs":{"total":289910292480,"available":269047005184,"internally_reserved":0,"allocated":15844249600,"data_stored":18737482542,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1508675,"internal_metadata":5017500349},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,11,9,22,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15212544,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4452,"num_read_kb":17740,"num_write":4417,"num_write_kb":17668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001979"},"pg_stats":[{"pgid":"2.7","version":"92'13480","reported_seq":18509,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:05.813606+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:17:05.813606+0000","last_peered":"2023-10-21T05:17:05.813606+0000","last_clean":"2023-10-21T05:17:05.813606+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:17:05.813606+0000","last_undegraded":"2023-10-21T05:17:05.813606+0000","last_fullsized":"2023-10-21T05:17:05.813606+0000","mapping_epoch":13,"log_start":"76'3400","ondisk_log_start":"76'3400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'6876","reported_seq":10431,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:05.790942+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:17:05.790942+0000","last_peered":"2023-10-21T05:17:05.790942+0000","last_clean":"2023-10-21T05:17:05.790942+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:17:05.790942+0000","last_undegraded":"2023-10-21T05:17:05.790942+0000","last_fullsized":"2023-10-21T05:17:05.790942+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6876,"log_dups_size":0,"ondisk_log_size":6876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"92'3446","reported_seq":4865,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:04.889282+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:17:04.889282+0000","last_peered":"2023-10-21T05:17:04.889282+0000","last_clean":"2023-10-21T05:17:04.889282+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:17:04.889282+0000","last_undegraded":"2023-10-21T05:17:04.889282+0000","last_fullsized":"2023-10-21T05:17:04.889282+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3446,"log_dups_size":0,"ondisk_log_size":3446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"92'6823","reported_seq":9788,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:05.787380+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:17:05.787380+0000","last_peered":"2023-10-21T05:17:05.787380+0000","last_clean":"2023-10-21T05:17:05.787380+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:17:05.787380+0000","last_undegraded":"2023-10-21T05:17:05.787380+0000","last_fullsized":"2023-10-21T05:17:05.787380+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6823,"log_dups_size":0,"ondisk_log_size":6823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"92'494400","reported_seq":983824,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:12.934599+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:17:12.934599+0000","last_peered":"2023-10-21T05:17:12.934599+0000","last_clean":"2023-10-21T05:17:12.934599+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:17:12.934599+0000","last_undegraded":"2023-10-21T05:17:12.934599+0000","last_fullsized":"2023-10-21T05:17:12.934599+0000","mapping_epoch":16,"log_start":"91'484318","ondisk_log_start":"91'484318","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2323,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 573s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6903611392,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":490593,"num_read_kb":1931828,"num_write":491316,"num_write_kb":1965264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":221,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:24.905437+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:16:24.905437+0000","last_peered":"2023-10-21T05:16:24.905437+0000","last_clean":"2023-10-21T05:16:24.905437+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:16:24.905437+0000","last_undegraded":"2023-10-21T05:16:24.905437+0000","last_fullsized":"2023-10-21T05:16:24.905437+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"92'498246","reported_seq":993557,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:09.364798+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:17:09.364798+0000","last_peered":"2023-10-21T05:17:09.364798+0000","last_clean":"2023-10-21T05:17:09.364798+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:17:09.364798+0000","last_undegraded":"2023-10-21T05:17:09.364798+0000","last_fullsized":"2023-10-21T05:17:09.364798+0000","mapping_epoch":16,"log_start":"91'488218","ondisk_log_start":"91'488218","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2235,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 558s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7019008000,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":496556,"num_read_kb":1954988,"num_write":495111,"num_write_kb":1980444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3922,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:09.835742+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:17:09.835742+0000","last_peered":"2023-10-21T05:17:09.835742+0000","last_clean":"2023-10-21T05:17:09.835742+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:17:09.835742+0000","last_undegraded":"2023-10-21T05:17:09.835742+0000","last_fullsized":"2023-10-21T05:17:09.835742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"92'466950","reported_seq":931483,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:12.930973+0000","last_change":"2023-10-21T05:17:09.881147+0000","last_active":"2023-10-21T05:17:12.930973+0000","last_peered":"2023-10-21T05:17:12.930973+0000","last_clean":"2023-10-21T05:17:12.930973+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:17:12.930973+0000","last_undegraded":"2023-10-21T05:17:12.930973+0000","last_fullsized":"2023-10-21T05:17:12.930973+0000","mapping_epoch":16,"log_start":"91'456920","ondisk_log_start":"91'456920","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2779,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"queued for deep scrub","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6502883328,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":465522,"num_read_kb":1832664,"num_write":463708,"num_write_kb":1854832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3857,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:09.833348+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:17:09.833348+0000","last_peered":"2023-10-21T05:17:09.833348+0000","last_clean":"2023-10-21T05:17:09.833348+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:17:09.833348+0000","last_undegraded":"2023-10-21T05:17:09.833348+0000","last_fullsized":"2023-10-21T05:17:09.833348+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"92'485108","reported_seq":966745,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:09.348935+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:17:09.348935+0000","last_peered":"2023-10-21T05:17:09.348935+0000","last_clean":"2023-10-21T05:17:09.348935+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:17:09.348935+0000","last_undegraded":"2023-10-21T05:17:09.348935+0000","last_fullsized":"2023-10-21T05:17:09.348935+0000","mapping_epoch":16,"log_start":"91'475024","ondisk_log_start":"91'475024","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":129,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 35s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6827249683,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":482919,"num_read_kb":1900888,"num_write":483334,"num_write_kb":1933330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'16821","reported_seq":23201,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:05.840144+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:17:05.840144+0000","last_peered":"2023-10-21T05:17:05.840144+0000","last_clean":"2023-10-21T05:17:05.840144+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:17:05.840144+0000","last_undegraded":"2023-10-21T05:17:05.840144+0000","last_fullsized":"2023-10-21T05:17:05.840144+0000","mapping_epoch":13,"log_start":"80'6800","ondisk_log_start":"80'6800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":266,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:58.808976+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:15:58.808976+0000","last_peered":"2023-10-21T05:15:58.808976+0000","last_clean":"2023-10-21T05:15:58.808976+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:15:58.808976+0000","last_undegraded":"2023-10-21T05:15:58.808976+0000","last_fullsized":"2023-10-21T05:15:58.808976+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27252752403,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1935590,"num_read_kb":7620368,"num_write":1933469,"num_write_kb":7733870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15843000320,"data_stored":18736971814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40224,"ondisk_log_size":40224,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37564,"ondisk_log_size":37564,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739280,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9733612,"kb_used_data":7737568,"kb_used_omap":786,"kb_used_meta":1995245,"kb_avail":84638228,"statfs":{"total":96636764160,"available":86669545472,"internally_reserved":0,"allocated":7923269632,"data_stored":9369730963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805125,"internal_metadata":2043131643},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:00 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.54100000000000004,"15min":0.48999999999999999},"min":{"1min":0.19,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":4.7560000000000002,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.59799999999999998,"15min":0.49199999999999999},"min":{"1min":0.21199999999999999,"5min":0.192,"15min":0.185},"max":{"1min":4.8449999999999998,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.50700000000000001,"15min":0.58299999999999996},"min":{"1min":0.23400000000000001,"5min":0.188,"15min":0.188},"max":{"1min":2.6299999999999999,"5min":5.2380000000000004,"15min":7.8449999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.55700000000000005,"15min":0.52700000000000002},"min":{"1min":0.23699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.8010000000000002,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.46200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739276,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5402440,"kb_used_data":3914432,"kb_used_omap":355,"kb_used_meta":1487644,"kb_avail":88969400,"statfs":{"total":96636764160,"available":91104665600,"internally_reserved":0,"allocated":4008378368,"data_stored":4754270181,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":1523348040},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,3,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.41999999999999998,"15min":0.48399999999999999},"min":{"1min":0.19900000000000001,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":1.129,"5min":2.0609999999999999,"15min":4.5599999999999996},"last":0.37},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.40600000000000003,"15min":0.45600000000000002},"min":{"1min":0.249,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":1.153,"5min":2.0270000000000001,"15min":4.6109999999999998},"last":0.32100000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.48799999999999999,"15min":0.45300000000000001},"min":{"1min":0.214,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.67200000000000004,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.26300000000000001,"5min":0.247,"15min":0.20499999999999999},"max":{"1min":0.70299999999999996,"5min":4.1920000000000002,"15min":4.306},"last":0.35599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5238252,"kb_used_data":3820900,"kb_used_omap":331,"kb_used_meta":1417012,"kb_avail":89133588,"statfs":{"total":96636764160,"available":91272794112,"internally_reserved":0,"allocated":3912601600,"data_stored":4613481398,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":339590,"internal_metadata":1451020666},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,13,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42599999999999999,"15min":0.38400000000000001},"min":{"1min":0.218,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":1.2390000000000001,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":0.38},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48399999999999999,"15min":0.44},"min":{"1min":0.28100000000000003,"5min":0.247,"15min":0.217},"max":{"1min":1.2769999999999999,"5min":5.0469999999999997,"15min":5.0469999999999997},"last":0.41699999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.54200000000000004,"15min":0.46999999999999997},"min":{"1min":0.253,"5min":0.221,"15min":0.22},"max":{"1min":2.3809999999999998,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55700000000000005,"15min":0.51600000000000001},"min":{"1min":0.27400000000000002,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":2.536,"5min":4.1340000000000003,"15min":6.3710000000000004},"last":0.48299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3911778304,"data_stored":4612714496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4007718912,"data_stored":4753768467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7923503104,"data_stored":9370488851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:17:15.374 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:15.374 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:15.375 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:15.375 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:35.376 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:17:35.931 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:17:35.932 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:17:35.987 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2374,"stamp":"2023-10-21T05:17:34.194515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27277597825,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1961826,"num_read_kb":7665153,"num_write":1941247,"num_write_kb":7764231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77812,"ondisk_log_size":77812,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20107500,"kb_used_data":15529428,"kb_used_omap":1474,"kb_used_meta":4576573,"kb_avail":263008020,"statfs":{"total":289910292480,"available":269320212480,"internally_reserved":0,"allocated":15902134272,"data_stored":18795358154,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1510355,"internal_metadata":4686410797},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,6,16,16,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12140544,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3605,"num_read_kb":14344,"num_write":3529,"num_write_kb":14116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002323"},"pg_stats":[{"pgid":"2.7","version":"92'13516","reported_seq":18561,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:27.327745+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:17:27.327745+0000","last_peered":"2023-10-21T05:17:27.327745+0000","last_clean":"2023-10-21T05:17:27.327745+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:17:27.327745+0000","last_undegraded":"2023-10-21T05:17:27.327745+0000","last_fullsized":"2023-10-21T05:17:27.327745+0000","mapping_epoch":13,"log_start":"76'3500","ondisk_log_start":"76'3500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'6894","reported_seq":10457,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:27.302232+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:17:27.302232+0000","last_peered":"2023-10-21T05:17:27.302232+0000","last_clean":"2023-10-21T05:17:27.302232+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:17:27.302232+0000","last_undegraded":"2023-10-21T05:17:27.302232+0000","last_fullsized":"2023-10-21T05:17:27.302232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6894,"log_dups_size":0,"ondisk_log_size":6894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"92'3455","reported_seq":4878,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:26.476719+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:17:26.476719+0000","last_peered":"2023-10-21T05:17:26.476719+0000","last_clean":"2023-10-21T05:17:26.476719+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:17:26.476719+0000","last_undegraded":"2023-10-21T05:17:26.476719+0000","last_fullsized":"2023-10-21T05:17:26.476719+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3455,"log_dups_size":0,"ondisk_log_size":3455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"92'6841","reported_seq":9814,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:27.302102+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:17:27.302102+0000","last_peered":"2023-10-21T05:17:27.302102+0000","last_clean":"2023-10-21T05:17:27.302102+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:17:27.302102+0000","last_undegraded":"2023-10-21T05:17:27.302102+0000","last_fullsized":"2023-10-21T05:17:27.302102+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6841,"log_dups_size":0,"ondisk_log_size":6841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'496244","reported_seq":987488,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:33.074184+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:17:33.074184+0000","last_peered":"2023-10-21T05:17:33.074184+0000","last_clean":"2023-10-21T05:17:33.074184+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:17:33.074184+0000","last_undegraded":"2023-10-21T05:17:33.074184+0000","last_fullsized":"2023-10-21T05:17:33.074184+0000","mapping_epoch":16,"log_start":"91'486218","ondisk_log_start":"91'486218","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2376,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 593s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6909734912,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":492394,"num_read_kb":1938980,"num_write":493130,"num_write_kb":1972520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":222,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:02.888470+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:17:02.888470+0000","last_peered":"2023-10-21T05:17:02.888470+0000","last_clean":"2023-10-21T05:17:02.888470+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:17:02.888470+0000","last_undegraded":"2023-10-21T05:17:02.888470+0000","last_fullsized":"2023-10-21T05:17:02.888470+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"92'500095","reported_seq":997313,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:29.499828+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:17:29.499828+0000","last_peered":"2023-10-21T05:17:29.499828+0000","last_clean":"2023-10-21T05:17:29.499828+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:17:29.499828+0000","last_undegraded":"2023-10-21T05:17:29.499828+0000","last_fullsized":"2023-10-21T05:17:29.499828+0000","mapping_epoch":16,"log_start":"91'490018","ondisk_log_start":"91'490018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2326,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 578s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7025291264,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":498454,"num_read_kb":1962556,"num_write":496904,"num_write_kb":1987616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3935,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:30.315652+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:17:30.315652+0000","last_peered":"2023-10-21T05:17:30.315652+0000","last_clean":"2023-10-21T05:17:30.315652+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:17:30.315652+0000","last_undegraded":"2023-10-21T05:17:30.315652+0000","last_fullsized":"2023-10-21T05:17:30.315652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'468627","reported_seq":934881,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:33.060438+0000","last_change":"2023-10-21T05:17:32.988732+0000","last_active":"2023-10-21T05:17:33.060438+0000","last_peered":"2023-10-21T05:17:33.060438+0000","last_clean":"2023-10-21T05:17:33.060438+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:17:33.060438+0000","last_undegraded":"2023-10-21T05:17:33.060438+0000","last_fullsized":"2023-10-21T05:17:33.060438+0000","mapping_epoch":16,"log_start":"91'458620","ondisk_log_start":"91'458620","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2779,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"queued for deep scrub","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6508527616,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":467237,"num_read_kb":1839516,"num_write":465385,"num_write_kb":1861540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3870,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:30.442475+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:17:30.442475+0000","last_peered":"2023-10-21T05:17:30.442475+0000","last_clean":"2023-10-21T05:17:30.442475+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:17:30.442475+0000","last_undegraded":"2023-10-21T05:17:30.442475+0000","last_fullsized":"2023-10-21T05:17:30.442475+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"92'486937","reported_seq":970324,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:29.507176+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:17:29.507176+0000","last_peered":"2023-10-21T05:17:29.507176+0000","last_clean":"2023-10-21T05:17:29.507176+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:17:29.507176+0000","last_undegraded":"2023-10-21T05:17:29.507176+0000","last_fullsized":"2023-10-21T05:17:29.507176+0000","mapping_epoch":16,"log_start":"91'476924","ondisk_log_start":"91'476924","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":252,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 56s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6833451027,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":484653,"num_read_kb":1907780,"num_write":485090,"num_write_kb":1940354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'16866","reported_seq":23266,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:27.346361+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:17:27.346361+0000","last_peered":"2023-10-21T05:17:27.346361+0000","last_clean":"2023-10-21T05:17:27.346361+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:17:27.346361+0000","last_undegraded":"2023-10-21T05:17:27.346361+0000","last_fullsized":"2023-10-21T05:17:27.346361+0000","mapping_epoch":13,"log_start":"80'6800","ondisk_log_start":"80'6800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":266,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-21T05:15:58.808976+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:15:58.808976+0000","last_peered":"2023-10-21T05:15:58.808976+0000","last_clean":"2023-10-21T05:15:58.808976+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:15:58.808976+0000","last_undegraded":"2023-10-21T05:15:58.808976+0000","last_fullsized":"2023-10-21T05:15:58.808976+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27277004819,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1942738,"num_read_kb":7648832,"num_write":1940509,"num_write_kb":7762030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15900921856,"data_stored":18794893350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40123,"ondisk_log_size":40123,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37590,"ondisk_log_size":37590,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739284,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9341400,"kb_used_data":7765968,"kb_used_omap":787,"kb_used_meta":1574636,"kb_avail":85030440,"statfs":{"total":96636764160,"available":87071170560,"internally_reserved":0,"allocated":7952351232,"data_stored":9398808033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":806805,"internal_metadata":1612427371},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:00 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.54100000000000004,"15min":0.48999999999999999},"min":{"1min":0.19,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":4.7560000000000002,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.59799999999999998,"15min":0.49199999999999999},"min":{"1min":0.21199999999999999,"5min":0.192,"15min":0.185},"max":{"1min":4.8449999999999998,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.49399999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.50700000000000001,"15min":0.58299999999999996},"min":{"1min":0.23400000000000001,"5min":0.188,"15min":0.188},"max":{"1min":2.6299999999999999,"5min":5.2380000000000004,"15min":7.8449999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.55700000000000005,"15min":0.52700000000000002},"min":{"1min":0.23699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.8010000000000002,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.318}]}]},{"osd":1,"up_from":8,"seq":34359739280,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5452664,"kb_used_data":3928560,"kb_used_omap":355,"kb_used_meta":1523740,"kb_avail":88919176,"statfs":{"total":96636764160,"available":91053236224,"internally_reserved":0,"allocated":4022845440,"data_stored":4768737253,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":1560310344},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,10,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.41999999999999998,"15min":0.48399999999999999},"min":{"1min":0.19900000000000001,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":1.129,"5min":2.0609999999999999,"15min":4.5599999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.40600000000000003,"15min":0.45600000000000002},"min":{"1min":0.249,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":1.153,"5min":2.0270000000000001,"15min":4.6109999999999998},"last":0.49299999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.48799999999999999,"15min":0.45300000000000001},"min":{"1min":0.214,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.67200000000000004,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.26300000000000001,"5min":0.247,"15min":0.20499999999999999},"max":{"1min":0.70299999999999996,"5min":4.1920000000000002,"15min":4.306},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359739285,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5313436,"kb_used_data":3834900,"kb_used_omap":331,"kb_used_meta":1478196,"kb_avail":89058404,"statfs":{"total":96636764160,"available":91195805696,"internally_reserved":0,"allocated":3926937600,"data_stored":4627812868,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":339590,"internal_metadata":1513673082},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42599999999999999,"15min":0.38400000000000001},"min":{"1min":0.218,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":1.2390000000000001,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48399999999999999,"15min":0.44},"min":{"1min":0.28100000000000003,"5min":0.247,"15min":0.217},"max":{"1min":1.2769999999999999,"5min":5.0469999999999997,"15min":5.0469999999999997},"last":0.39200000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.54200000000000004,"15min":0.46999999999999997},"min":{"1min":0.253,"5min":0.221,"15min":0.22},"max":{"1min":2.3809999999999998,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55700000000000005,"15min":0.51600000000000001},"min":{"1min":0.27400000000000002,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":2.536,"5min":4.1340000000000003,"15min":6.3710000000000004},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3926085632,"data_stored":4627021824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4022267904,"data_stored":4768317459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7952568320,"data_stored":9399554067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:17:35.989 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:35.989 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:35.990 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:35.990 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:17:55.992 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:17:56.548 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:17:56.550 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:17:56.589 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2385,"stamp":"2023-10-21T05:17:56.198274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27305254017,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1969714,"num_read_kb":7696525,"num_write":1949234,"num_write_kb":7796179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77923,"ondisk_log_size":77923,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20324328,"kb_used_data":15589328,"kb_used_omap":1474,"kb_used_meta":4733501,"kb_avail":262791192,"statfs":{"total":289910292480,"available":269098180608,"internally_reserved":0,"allocated":15963471872,"data_stored":18856691224,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1510355,"internal_metadata":4847105069},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,4,10,11,20,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15474688,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4404,"num_read_kb":17524,"num_write":4453,"num_write_kb":17812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001996"},"pg_stats":[{"pgid":"2.7","version":"93'13560","reported_seq":18627,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:54.194014+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:17:54.194014+0000","last_peered":"2023-10-21T05:17:54.194014+0000","last_clean":"2023-10-21T05:17:54.194014+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:17:54.194014+0000","last_undegraded":"2023-10-21T05:17:54.194014+0000","last_fullsized":"2023-10-21T05:17:54.194014+0000","mapping_epoch":13,"log_start":"76'3500","ondisk_log_start":"76'3500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'6916","reported_seq":10491,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:54.175282+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:17:54.175282+0000","last_peered":"2023-10-21T05:17:54.175282+0000","last_clean":"2023-10-21T05:17:54.175282+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:17:54.175282+0000","last_undegraded":"2023-10-21T05:17:54.175282+0000","last_fullsized":"2023-10-21T05:17:54.175282+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6916,"log_dups_size":0,"ondisk_log_size":6916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"93'3466","reported_seq":4895,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:50.792251+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:17:50.792251+0000","last_peered":"2023-10-21T05:17:50.792251+0000","last_clean":"2023-10-21T05:17:50.792251+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:17:50.792251+0000","last_undegraded":"2023-10-21T05:17:50.792251+0000","last_fullsized":"2023-10-21T05:17:50.792251+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3466,"log_dups_size":0,"ondisk_log_size":3466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"93'6863","reported_seq":9847,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:54.175803+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:17:54.175803+0000","last_peered":"2023-10-21T05:17:54.175803+0000","last_clean":"2023-10-21T05:17:54.175803+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:17:54.175803+0000","last_undegraded":"2023-10-21T05:17:54.175803+0000","last_fullsized":"2023-10-21T05:17:54.175803+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6863,"log_dups_size":0,"ondisk_log_size":6863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'498119","reported_seq":991139,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:53.200694+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:17:53.200694+0000","last_peered":"2023-10-21T05:17:53.200694+0000","last_clean":"2023-10-21T05:17:53.200694+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:17:53.200694+0000","last_undegraded":"2023-10-21T05:17:53.200694+0000","last_fullsized":"2023-10-21T05:17:53.200694+0000","mapping_epoch":16,"log_start":"91'488118","ondisk_log_start":"91'488118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2425,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 613s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6915973120,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":494160,"num_read_kb":1946008,"num_write":494973,"num_write_kb":1979892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":222,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:02.888470+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:17:02.888470+0000","last_peered":"2023-10-21T05:17:02.888470+0000","last_clean":"2023-10-21T05:17:02.888470+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:17:02.888470+0000","last_undegraded":"2023-10-21T05:17:02.888470+0000","last_fullsized":"2023-10-21T05:17:02.888470+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"93'502411","reported_seq":1001926,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:54.663502+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:17:54.663502+0000","last_peered":"2023-10-21T05:17:54.663502+0000","last_clean":"2023-10-21T05:17:54.663502+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:17:54.663502+0000","last_undegraded":"2023-10-21T05:17:54.663502+0000","last_fullsized":"2023-10-21T05:17:54.663502+0000","mapping_epoch":16,"log_start":"91'492318","ondisk_log_start":"91'492318","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2439,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 604s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7033319424,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":500731,"num_read_kb":1971608,"num_write":499156,"num_write_kb":1996624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3947,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:50.793770+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:17:50.793770+0000","last_peered":"2023-10-21T05:17:50.793770+0000","last_clean":"2023-10-21T05:17:50.793770+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:17:50.793770+0000","last_undegraded":"2023-10-21T05:17:50.793770+0000","last_fullsized":"2023-10-21T05:17:50.793770+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'470322","reported_seq":938283,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:53.201195+0000","last_change":"2023-10-21T05:17:47.919391+0000","last_active":"2023-10-21T05:17:53.201195+0000","last_peered":"2023-10-21T05:17:53.201195+0000","last_clean":"2023-10-21T05:17:53.201195+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:17:53.201195+0000","last_undegraded":"2023-10-21T05:17:53.201195+0000","last_fullsized":"2023-10-21T05:17:53.201195+0000","mapping_epoch":16,"log_start":"91'460320","ondisk_log_start":"91'460320","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2779,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"queued for deep scrub","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6514294784,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":468942,"num_read_kb":1846300,"num_write":467080,"num_write_kb":1868320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3882,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:50.804398+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:17:50.804398+0000","last_peered":"2023-10-21T05:17:50.804398+0000","last_clean":"2023-10-21T05:17:50.804398+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:17:50.804398+0000","last_undegraded":"2023-10-21T05:17:50.804398+0000","last_fullsized":"2023-10-21T05:17:50.804398+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'489208","reported_seq":974750,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:17:54.629651+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:17:54.629651+0000","last_peered":"2023-10-21T05:17:54.629651+0000","last_clean":"2023-10-21T05:17:54.629651+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:17:54.629651+0000","last_undegraded":"2023-10-21T05:17:54.629651+0000","last_fullsized":"2023-10-21T05:17:54.629651+0000","mapping_epoch":16,"log_start":"91'479124","ondisk_log_start":"91'479124","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":370,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 81s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6841073683,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":486793,"num_read_kb":1916288,"num_write":487287,"num_write_kb":1949142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'16921","reported_seq":23347,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:54.243416+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:17:54.243416+0000","last_peered":"2023-10-21T05:17:54.243416+0000","last_clean":"2023-10-21T05:17:54.243416+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:17:54.243416+0000","last_undegraded":"2023-10-21T05:17:54.243416+0000","last_fullsized":"2023-10-21T05:17:54.243416+0000","mapping_epoch":13,"log_start":"80'6900","ondisk_log_start":"80'6900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":267,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:16:24.910543+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:16:24.910543+0000","last_peered":"2023-10-21T05:16:24.910543+0000","last_clean":"2023-10-21T05:16:24.910543+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:16:24.910543+0000","last_undegraded":"2023-10-21T05:16:24.910543+0000","last_fullsized":"2023-10-21T05:16:24.910543+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27304661011,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1950626,"num_read_kb":7680204,"num_write":1948496,"num_write_kb":7793978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15962550272,"data_stored":18856521766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40180,"ondisk_log_size":40180,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37644,"ondisk_log_size":37644,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739288,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9451264,"kb_used_data":7793912,"kb_used_omap":787,"kb_used_meta":1656556,"kb_avail":84920576,"statfs":{"total":96636764160,"available":86958669824,"internally_reserved":0,"allocated":7980965888,"data_stored":9427422689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":806805,"internal_metadata":1696313451},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:00 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.54100000000000004,"15min":0.48999999999999999},"min":{"1min":0.19,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":4.7560000000000002,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.59799999999999998,"15min":0.49199999999999999},"min":{"1min":0.21199999999999999,"5min":0.192,"15min":0.185},"max":{"1min":4.8449999999999998,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.26600000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.50700000000000001,"15min":0.58299999999999996},"min":{"1min":0.23400000000000001,"5min":0.188,"15min":0.188},"max":{"1min":2.6299999999999999,"5min":5.2380000000000004,"15min":7.8449999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.55700000000000005,"15min":0.52700000000000002},"min":{"1min":0.23699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.8010000000000002,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.23100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739285,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5531644,"kb_used_data":3946420,"kb_used_omap":355,"kb_used_meta":1584860,"kb_avail":88840196,"statfs":{"total":96636764160,"available":90972360704,"internally_reserved":0,"allocated":4041134080,"data_stored":4787021363,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":1622897224},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,7,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.41999999999999998,"15min":0.48399999999999999},"min":{"1min":0.19900000000000001,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":1.129,"5min":2.0609999999999999,"15min":4.5599999999999996},"last":0.311},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.40600000000000003,"15min":0.45600000000000002},"min":{"1min":0.249,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":1.153,"5min":2.0270000000000001,"15min":4.6109999999999998},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:17:02 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.48799999999999999,"15min":0.45300000000000001},"min":{"1min":0.214,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.67200000000000004,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.26300000000000001,"5min":0.247,"15min":0.20499999999999999},"max":{"1min":0.70299999999999996,"5min":4.1920000000000002,"15min":4.306},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739289,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5341420,"kb_used_data":3848996,"kb_used_omap":331,"kb_used_meta":1492084,"kb_avail":89030420,"statfs":{"total":96636764160,"available":91167150080,"internally_reserved":0,"allocated":3941371904,"data_stored":4642247172,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":339590,"internal_metadata":1527894394},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,0,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42599999999999999,"15min":0.38400000000000001},"min":{"1min":0.218,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":1.2390000000000001,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48399999999999999,"15min":0.44},"min":{"1min":0.28100000000000003,"5min":0.247,"15min":0.217},"max":{"1min":1.2769999999999999,"5min":5.0469999999999997,"15min":5.0469999999999997},"last":0.40699999999999997}]},{"osd":2,"last update":"Sat Oct 21 05:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.54200000000000004,"15min":0.46999999999999997},"min":{"1min":0.253,"5min":0.221,"15min":0.22},"max":{"1min":2.3809999999999998,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55700000000000005,"15min":0.51600000000000001},"min":{"1min":0.27400000000000002,"5min":0.251,"15min":0.23499999999999999},"max":{"1min":2.536,"5min":4.1340000000000003,"15min":6.3710000000000004},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3940577280,"data_stored":4641513472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4040482816,"data_stored":4786532371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7981490176,"data_stored":9428475923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:17:56.591 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:56.592 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:56.592 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:17:56.593 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:18:16.594 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:18:17.250 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:18:17.253 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:18:17.285 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2395,"stamp":"2023-10-21T05:18:16.203191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27329678465,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1976459,"num_read_kb":7723389,"num_write":1956366,"num_write_kb":7824707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78146,"ondisk_log_size":78146,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20567480,"kb_used_data":15646368,"kb_used_omap":1474,"kb_used_meta":4919613,"kb_avail":262548040,"statfs":{"total":289910292480,"available":268849192960,"internally_reserved":0,"allocated":16021880832,"data_stored":18915100184,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1510355,"internal_metadata":5037683757},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,9,15,10,16,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15237120,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4248,"num_read_kb":16924,"num_write":4451,"num_write_kb":17804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002554"},"pg_stats":[{"pgid":"2.7","version":"93'13608","reported_seq":18691,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:14.131880+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:18:14.131880+0000","last_peered":"2023-10-21T05:18:14.131880+0000","last_clean":"2023-10-21T05:18:14.131880+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:18:14.131880+0000","last_undegraded":"2023-10-21T05:18:14.131880+0000","last_fullsized":"2023-10-21T05:18:14.131880+0000","mapping_epoch":13,"log_start":"78'3600","ondisk_log_start":"78'3600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'6940","reported_seq":10523,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:14.110665+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:18:14.110665+0000","last_peered":"2023-10-21T05:18:14.110665+0000","last_clean":"2023-10-21T05:18:14.110665+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:18:14.110665+0000","last_undegraded":"2023-10-21T05:18:14.110665+0000","last_fullsized":"2023-10-21T05:18:14.110665+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6940,"log_dups_size":0,"ondisk_log_size":6940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"93'3478","reported_seq":4911,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:13.904663+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:18:13.904663+0000","last_peered":"2023-10-21T05:18:13.904663+0000","last_clean":"2023-10-21T05:18:13.904663+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:18:13.904663+0000","last_undegraded":"2023-10-21T05:18:13.904663+0000","last_fullsized":"2023-10-21T05:18:13.904663+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"93'6887","reported_seq":9879,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:14.107749+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:18:14.107749+0000","last_peered":"2023-10-21T05:18:14.107749+0000","last_clean":"2023-10-21T05:18:14.107749+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:18:14.107749+0000","last_undegraded":"2023-10-21T05:18:14.107749+0000","last_fullsized":"2023-10-21T05:18:14.107749+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6887,"log_dups_size":0,"ondisk_log_size":6887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'500000","reported_seq":994686,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:13.325957+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:18:13.325957+0000","last_peered":"2023-10-21T05:18:13.325957+0000","last_clean":"2023-10-21T05:18:13.325957+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:18:13.325957+0000","last_undegraded":"2023-10-21T05:18:13.325957+0000","last_fullsized":"2023-10-21T05:18:13.325957+0000","mapping_epoch":16,"log_start":"91'489918","ondisk_log_start":"91'489918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2493,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 633s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6922190848,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":495817,"num_read_kb":1952596,"num_write":496813,"num_write_kb":1987252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":222,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:02.888470+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:17:02.888470+0000","last_peered":"2023-10-21T05:17:02.888470+0000","last_clean":"2023-10-21T05:17:02.888470+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:17:02.888470+0000","last_undegraded":"2023-10-21T05:17:02.888470+0000","last_fullsized":"2023-10-21T05:17:02.888470+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"93'504297","reported_seq":1005592,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:14.808806+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:18:14.808806+0000","last_peered":"2023-10-21T05:18:14.808806+0000","last_clean":"2023-10-21T05:18:14.808806+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:18:14.808806+0000","last_undegraded":"2023-10-21T05:18:14.808806+0000","last_fullsized":"2023-10-21T05:18:14.808806+0000","mapping_epoch":16,"log_start":"92'494219","ondisk_log_start":"92'494219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2507,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 624s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7039823872,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":502502,"num_read_kb":1978668,"num_write":501000,"num_write_kb":2004000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3959,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:11.272886+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:18:11.272886+0000","last_peered":"2023-10-21T05:18:11.272886+0000","last_clean":"2023-10-21T05:18:11.272886+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:18:11.272886+0000","last_undegraded":"2023-10-21T05:18:11.272886+0000","last_fullsized":"2023-10-21T05:18:11.272886+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'472001","reported_seq":941542,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:13.316815+0000","last_change":"2023-10-21T05:18:13.221696+0000","last_active":"2023-10-21T05:18:13.316815+0000","last_peered":"2023-10-21T05:18:13.316815+0000","last_clean":"2023-10-21T05:18:13.316815+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:18:13.316815+0000","last_undegraded":"2023-10-21T05:18:13.316815+0000","last_fullsized":"2023-10-21T05:18:13.316815+0000","mapping_epoch":16,"log_start":"91'461920","ondisk_log_start":"91'461920","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2779,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"queued for deep scrub","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6519767040,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":470519,"num_read_kb":1852584,"num_write":468759,"num_write_kb":1875036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3895,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:11.274546+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:18:11.274546+0000","last_peered":"2023-10-21T05:18:11.274546+0000","last_clean":"2023-10-21T05:18:11.274546+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:18:11.274546+0000","last_undegraded":"2023-10-21T05:18:11.274546+0000","last_fullsized":"2023-10-21T05:18:11.274546+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'491018","reported_seq":978309,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:14.830220+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:18:14.830220+0000","last_peered":"2023-10-21T05:18:14.830220+0000","last_clean":"2023-10-21T05:18:14.830220+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:18:14.830220+0000","last_undegraded":"2023-10-21T05:18:14.830220+0000","last_fullsized":"2023-10-21T05:18:14.830220+0000","mapping_epoch":16,"log_start":"91'480924","ondisk_log_start":"91'480924","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":439,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 101s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6847303699,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":488533,"num_read_kb":1923220,"num_write":489056,"num_write_kb":1956218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'16981","reported_seq":23427,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:14.172106+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:18:14.172106+0000","last_peered":"2023-10-21T05:18:14.172106+0000","last_clean":"2023-10-21T05:18:14.172106+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:18:14.172106+0000","last_undegraded":"2023-10-21T05:18:14.172106+0000","last_fullsized":"2023-10-21T05:18:14.172106+0000","mapping_epoch":13,"log_start":"80'6900","ondisk_log_start":"80'6900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":268,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:29.944118+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:17:29.944118+0000","last_peered":"2023-10-21T05:17:29.944118+0000","last_clean":"2023-10-21T05:17:29.944118+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:17:29.944118+0000","last_undegraded":"2023-10-21T05:17:29.944118+0000","last_fullsized":"2023-10-21T05:17:29.944118+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27329085459,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1957371,"num_read_kb":7707068,"num_write":1955628,"num_write_kb":7822506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16020779008,"data_stored":18914750502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40335,"ondisk_log_size":40335,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37712,"ondisk_log_size":37712,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739292,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9586708,"kb_used_data":7822476,"kb_used_omap":787,"kb_used_meta":1763436,"kb_avail":84785132,"statfs":{"total":96636764160,"available":86819975168,"internally_reserved":0,"allocated":8010215424,"data_stored":9456672225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":806805,"internal_metadata":1805758571},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:00 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52300000000000002,"15min":0.497},"min":{"1min":0.19800000000000001,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":2.379,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.628},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.57199999999999995,"15min":0.497},"min":{"1min":0.192,"5min":0.192,"15min":0.185},"max":{"1min":2.3969999999999998,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.56299999999999994}]},{"osd":1,"last update":"Sat Oct 21 05:18:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54600000000000004,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.188},"max":{"1min":3.9910000000000001,"5min":5.2380000000000004,"15min":7.8449999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.54900000000000004,"15min":0.52900000000000003},"min":{"1min":0.223,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.52200000000000002,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.60199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739289,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5564028,"kb_used_data":3960756,"kb_used_omap":355,"kb_used_meta":1602908,"kb_avail":88807812,"statfs":{"total":96636764160,"available":90939199488,"internally_reserved":0,"allocated":4055814144,"data_stored":4801701427,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":1641378376},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,7,6,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:03 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.48899999999999999,"15min":0.51100000000000001},"min":{"1min":0.22700000000000001,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":6.173,"5min":6.173,"15min":6.173},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.47499999999999998,"15min":0.48099999999999998},"min":{"1min":0.27700000000000002,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.48699999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:18:03 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.41099999999999998,"15min":0.45900000000000002},"min":{"1min":0.219,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.73,"5min":4.3259999999999996,"15min":8.7669999999999995},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.39900000000000002,"15min":0.40799999999999997},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":0.71399999999999997,"5min":4.1920000000000002,"15min":4.306},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739293,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5416744,"kb_used_data":3863136,"kb_used_omap":331,"kb_used_meta":1553268,"kb_avail":88955096,"statfs":{"total":96636764160,"available":91090018304,"internally_reserved":0,"allocated":3955851264,"data_stored":4656726532,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":339590,"internal_metadata":1590546810},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,4,4,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:18:11 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.54900000000000004,"15min":0.42399999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":15.727,"5min":15.727,"15min":15.727},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.60599999999999998,"15min":0.47999999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.217},"max":{"1min":15.757999999999999,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.39700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:18:11 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.51600000000000001,"15min":0.46800000000000003},"min":{"1min":0.221,"5min":0.221,"15min":0.22},"max":{"1min":0.49399999999999999,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.629,"15min":0.53900000000000003},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.23499999999999999},"max":{"1min":10.967000000000001,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.33100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3954991104,"data_stored":4655927296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4055281664,"data_stored":4801331219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8010506240,"data_stored":9457491987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:18:17.287 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:17.287 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:17.287 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:17.288 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:18:37.289 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:18:37.737 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:18:37.738 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:18:37.775 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2405,"stamp":"2023-10-21T05:18:36.208474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27354197121,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1983626,"num_read_kb":7751929,"num_write":1963558,"num_write_kb":7853475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77969,"ondisk_log_size":77969,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20808448,"kb_used_data":15705128,"kb_used_omap":1474,"kb_used_meta":5101821,"kb_avail":262307072,"statfs":{"total":289910292480,"available":268602441728,"internally_reserved":0,"allocated":16082051072,"data_stored":18975270424,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1510355,"internal_metadata":5224264749},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,5,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15540224,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4499,"num_read_kb":17912,"num_write":4526,"num_write_kb":18104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003871"},"pg_stats":[{"pgid":"2.7","version":"93'13641","reported_seq":18739,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:34.960436+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:18:34.960436+0000","last_peered":"2023-10-21T05:18:34.960436+0000","last_clean":"2023-10-21T05:18:34.960436+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:18:34.960436+0000","last_undegraded":"2023-10-21T05:18:34.960436+0000","last_fullsized":"2023-10-21T05:18:34.960436+0000","mapping_epoch":13,"log_start":"78'3600","ondisk_log_start":"78'3600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'6958","reported_seq":10549,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:34.943045+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:18:34.943045+0000","last_peered":"2023-10-21T05:18:34.943045+0000","last_clean":"2023-10-21T05:18:34.943045+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:18:34.943045+0000","last_undegraded":"2023-10-21T05:18:34.943045+0000","last_fullsized":"2023-10-21T05:18:34.943045+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6958,"log_dups_size":0,"ondisk_log_size":6958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"93'3487","reported_seq":4924,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:34.227748+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:18:34.227748+0000","last_peered":"2023-10-21T05:18:34.227748+0000","last_clean":"2023-10-21T05:18:34.227748+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:18:34.227748+0000","last_undegraded":"2023-10-21T05:18:34.227748+0000","last_fullsized":"2023-10-21T05:18:34.227748+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3487,"log_dups_size":0,"ondisk_log_size":3487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"93'6904","reported_seq":9904,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:34.947871+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:18:34.947871+0000","last_peered":"2023-10-21T05:18:34.947871+0000","last_clean":"2023-10-21T05:18:34.947871+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:18:34.947871+0000","last_undegraded":"2023-10-21T05:18:34.947871+0000","last_fullsized":"2023-10-21T05:18:34.947871+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6904,"log_dups_size":0,"ondisk_log_size":6904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'501848","reported_seq":998419,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:33.450915+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:18:33.450915+0000","last_peered":"2023-10-21T05:18:33.450915+0000","last_clean":"2023-10-21T05:18:33.450915+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:18:33.450915+0000","last_undegraded":"2023-10-21T05:18:33.450915+0000","last_fullsized":"2023-10-21T05:18:33.450915+0000","mapping_epoch":16,"log_start":"92'491818","ondisk_log_start":"92'491818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2545,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 653s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6928318464,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":497693,"num_read_kb":1960056,"num_write":498632,"num_write_kb":1994528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":222,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-21T05:17:02.888470+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:17:02.888470+0000","last_peered":"2023-10-21T05:17:02.888470+0000","last_clean":"2023-10-21T05:17:02.888470+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:17:02.888470+0000","last_undegraded":"2023-10-21T05:17:02.888470+0000","last_fullsized":"2023-10-21T05:17:02.888470+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"93'506181","reported_seq":1009325,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:34.944638+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:18:34.944638+0000","last_peered":"2023-10-21T05:18:34.944638+0000","last_clean":"2023-10-21T05:18:34.944638+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:18:34.944638+0000","last_undegraded":"2023-10-21T05:18:34.944638+0000","last_fullsized":"2023-10-21T05:18:34.944638+0000","mapping_epoch":16,"log_start":"92'496119","ondisk_log_start":"92'496119","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2583,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 644s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7046184960,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":504340,"num_read_kb":1985996,"num_write":502838,"num_write_kb":2011352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3971,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:31.754616+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:18:31.754616+0000","last_peered":"2023-10-21T05:18:31.754616+0000","last_clean":"2023-10-21T05:18:31.754616+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:18:31.754616+0000","last_undegraded":"2023-10-21T05:18:31.754616+0000","last_fullsized":"2023-10-21T05:18:31.754616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'473721","reported_seq":944898,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:33.433435+0000","last_change":"2023-10-21T05:18:33.135271+0000","last_active":"2023-10-21T05:18:33.433435+0000","last_peered":"2023-10-21T05:18:33.433435+0000","last_clean":"2023-10-21T05:18:33.433435+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:18:33.433435+0000","last_undegraded":"2023-10-21T05:18:33.433435+0000","last_fullsized":"2023-10-21T05:18:33.433435+0000","mapping_epoch":16,"log_start":"92'463720","ondisk_log_start":"92'463720","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2779,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"queued for deep scrub","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6525480960,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":472157,"num_read_kb":1859116,"num_write":470479,"num_write_kb":1881916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3907,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:31.752518+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:18:31.752518+0000","last_peered":"2023-10-21T05:18:31.752518+0000","last_clean":"2023-10-21T05:18:31.752518+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:18:31.752518+0000","last_undegraded":"2023-10-21T05:18:31.752518+0000","last_fullsized":"2023-10-21T05:18:31.752518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'492876","reported_seq":981992,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:34.934084+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:18:34.934084+0000","last_peered":"2023-10-21T05:18:34.934084+0000","last_clean":"2023-10-21T05:18:34.934084+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:18:34.934084+0000","last_undegraded":"2023-10-21T05:18:34.934084+0000","last_fullsized":"2023-10-21T05:18:34.934084+0000","mapping_epoch":16,"log_start":"92'482824","ondisk_log_start":"92'482824","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":509,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 121s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6853619731,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":490348,"num_read_kb":1930440,"num_write":490871,"num_write_kb":1963478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'17017","reported_seq":23483,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:18:34.292035+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:18:34.292035+0000","last_peered":"2023-10-21T05:18:34.292035+0000","last_clean":"2023-10-21T05:18:34.292035+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:18:34.292035+0000","last_undegraded":"2023-10-21T05:18:34.292035+0000","last_fullsized":"2023-10-21T05:18:34.292035+0000","mapping_epoch":13,"log_start":"80'7000","ondisk_log_start":"80'7000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":268,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:29.944118+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:17:29.944118+0000","last_peered":"2023-10-21T05:17:29.944118+0000","last_clean":"2023-10-21T05:17:29.944118+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:17:29.944118+0000","last_undegraded":"2023-10-21T05:17:29.944118+0000","last_fullsized":"2023-10-21T05:17:29.944118+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27353604115,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1964538,"num_read_kb":7735608,"num_write":1962820,"num_write_kb":7851274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16079867904,"data_stored":18973839398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40145,"ondisk_log_size":40145,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37725,"ondisk_log_size":37725,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9680480,"kb_used_data":7852376,"kb_used_omap":787,"kb_used_meta":1827308,"kb_avail":84691360,"statfs":{"total":96636764160,"available":86723952640,"internally_reserved":0,"allocated":8040833024,"data_stored":9487289825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":806805,"internal_metadata":1871163499},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:00 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52300000000000002,"15min":0.497},"min":{"1min":0.19800000000000001,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":2.379,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.57199999999999995,"15min":0.497},"min":{"1min":0.192,"5min":0.192,"15min":0.185},"max":{"1min":2.3969999999999998,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.27000000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:18:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54600000000000004,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.188},"max":{"1min":3.9910000000000001,"5min":5.2380000000000004,"15min":7.8449999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.54900000000000004,"15min":0.52900000000000003},"min":{"1min":0.223,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.52200000000000002,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.253}]}]},{"osd":1,"up_from":8,"seq":34359739293,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5658244,"kb_used_data":3975612,"kb_used_omap":355,"kb_used_meta":1682268,"kb_avail":88713596,"statfs":{"total":96636764160,"available":90842722304,"internally_reserved":0,"allocated":4071026688,"data_stored":4816913971,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":1722643016},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:03 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.48899999999999999,"15min":0.51100000000000001},"min":{"1min":0.22700000000000001,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":6.173,"5min":6.173,"15min":6.173},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.47499999999999998,"15min":0.48099999999999998},"min":{"1min":0.27700000000000002,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.28699999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:18:03 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.41099999999999998,"15min":0.45900000000000002},"min":{"1min":0.219,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.73,"5min":4.3259999999999996,"15min":8.7669999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.39900000000000002,"15min":0.40799999999999997},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":0.71399999999999997,"5min":4.1920000000000002,"15min":4.306},"last":0.25}]}]},{"osd":0,"up_from":8,"seq":34359739297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5469724,"kb_used_data":3877140,"kb_used_omap":331,"kb_used_meta":1592244,"kb_avail":88902116,"statfs":{"total":96636764160,"available":91035766784,"internally_reserved":0,"allocated":3970191360,"data_stored":4671066628,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":339590,"internal_metadata":1630458234},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,4,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:18:11 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.54900000000000004,"15min":0.42399999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":15.727,"5min":15.727,"15min":15.727},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.60599999999999998,"15min":0.47999999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.217},"max":{"1min":15.757999999999999,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.35599999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:18:11 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.51600000000000001,"15min":0.46800000000000003},"min":{"1min":0.221,"5min":0.221,"15min":0.22},"max":{"1min":0.49399999999999999,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.629,"15min":0.53900000000000003},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.23499999999999999},"max":{"1min":10.967000000000001,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3969482752,"data_stored":4670418944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4070232064,"data_stored":4816281619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8040153088,"data_stored":9487138835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:18:37.776 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:37.777 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:37.778 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:37.778 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:18:57.779 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:18:58.295 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:18:58.297 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:18:58.333 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2417,"stamp":"2023-10-21T05:18:58.212308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27378728065,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1990331,"num_read_kb":7778673,"num_write":1970710,"num_write_kb":7882083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78125,"ondisk_log_size":78125,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21052708,"kb_used_data":15761864,"kb_used_omap":1474,"kb_used_meta":5289341,"kb_avail":262062812,"statfs":{"total":289910292480,"available":268352319488,"internally_reserved":0,"allocated":16140148736,"data_stored":19033354369,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1510355,"internal_metadata":5416285229},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,7,7,13,8,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12390400,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3415,"num_read_kb":13620,"num_write":3623,"num_write_kb":14492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002153"},"pg_stats":[{"pgid":"2.7","version":"94'13689","reported_seq":18805,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:18:54.813097+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:18:54.813097+0000","last_peered":"2023-10-21T05:18:54.813097+0000","last_clean":"2023-10-21T05:18:54.813097+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:18:54.813097+0000","last_undegraded":"2023-10-21T05:18:54.813097+0000","last_fullsized":"2023-10-21T05:18:54.813097+0000","mapping_epoch":13,"log_start":"78'3600","ondisk_log_start":"78'3600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"94'6982","reported_seq":10582,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:18:54.801748+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:18:54.801748+0000","last_peered":"2023-10-21T05:18:54.801748+0000","last_clean":"2023-10-21T05:18:54.801748+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:18:54.801748+0000","last_undegraded":"2023-10-21T05:18:54.801748+0000","last_fullsized":"2023-10-21T05:18:54.801748+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":6982,"log_dups_size":0,"ondisk_log_size":6982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"94'3499","reported_seq":4941,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:18:53.939881+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:18:53.939881+0000","last_peered":"2023-10-21T05:18:53.939881+0000","last_clean":"2023-10-21T05:18:53.939881+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:18:53.939881+0000","last_undegraded":"2023-10-21T05:18:53.939881+0000","last_fullsized":"2023-10-21T05:18:53.939881+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3499,"log_dups_size":0,"ondisk_log_size":3499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"94'6928","reported_seq":9937,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:18:54.801880+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:18:54.801880+0000","last_peered":"2023-10-21T05:18:54.801880+0000","last_clean":"2023-10-21T05:18:54.801880+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:18:54.801880+0000","last_undegraded":"2023-10-21T05:18:54.801880+0000","last_fullsized":"2023-10-21T05:18:54.801880+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6928,"log_dups_size":0,"ondisk_log_size":6928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"94'503722","reported_seq":1001980,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:53.590911+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:18:53.590911+0000","last_peered":"2023-10-21T05:18:53.590911+0000","last_clean":"2023-10-21T05:18:53.590911+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:18:53.590911+0000","last_undegraded":"2023-10-21T05:18:53.590911+0000","last_fullsized":"2023-10-21T05:18:53.590911+0000","mapping_epoch":16,"log_start":"92'493718","ondisk_log_start":"92'493718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2593,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 673s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6934642688,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":499372,"num_read_kb":1966752,"num_write":500474,"num_write_kb":2001896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":223,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:29.940068+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:17:29.940068+0000","last_peered":"2023-10-21T05:17:29.940068+0000","last_clean":"2023-10-21T05:17:29.940068+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:17:29.940068+0000","last_undegraded":"2023-10-21T05:17:29.940068+0000","last_fullsized":"2023-10-21T05:17:29.940068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"94'508029","reported_seq":1012880,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:55.101737+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:18:55.101737+0000","last_peered":"2023-10-21T05:18:55.101737+0000","last_clean":"2023-10-21T05:18:55.101737+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:18:55.101737+0000","last_undegraded":"2023-10-21T05:18:55.101737+0000","last_fullsized":"2023-10-21T05:18:55.101737+0000","mapping_epoch":16,"log_start":"92'498019","ondisk_log_start":"92'498019","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2652,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 664s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7052492800,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":506039,"num_read_kb":1992776,"num_write":504642,"num_write_kb":2018568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":3985,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:18:52.232636+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:18:52.232636+0000","last_peered":"2023-10-21T05:18:52.232636+0000","last_clean":"2023-10-21T05:18:52.232636+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:18:52.232636+0000","last_undegraded":"2023-10-21T05:18:52.232636+0000","last_fullsized":"2023-10-21T05:18:52.232636+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"94'475469","reported_seq":948285,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:18:53.600194+0000","last_change":"2023-10-21T05:18:53.579318+0000","last_active":"2023-10-21T05:18:53.600194+0000","last_peered":"2023-10-21T05:18:53.600194+0000","last_clean":"2023-10-21T05:18:53.600194+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:18:53.600194+0000","last_undegraded":"2023-10-21T05:18:53.600194+0000","last_fullsized":"2023-10-21T05:18:53.600194+0000","mapping_epoch":16,"log_start":"92'465420","ondisk_log_start":"92'465420","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2779,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"queued for deep scrub","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6531391488,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":473779,"num_read_kb":1865580,"num_write":472227,"num_write_kb":1888908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3920,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:18:52.234700+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:18:52.234700+0000","last_peered":"2023-10-21T05:18:52.234700+0000","last_clean":"2023-10-21T05:18:52.234700+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:18:52.234700+0000","last_undegraded":"2023-10-21T05:18:52.234700+0000","last_fullsized":"2023-10-21T05:18:52.234700+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"94'494690","reported_seq":985522,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:18:55.113580+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:18:55.113580+0000","last_peered":"2023-10-21T05:18:55.113580+0000","last_clean":"2023-10-21T05:18:55.113580+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:18:55.113580+0000","last_undegraded":"2023-10-21T05:18:55.113580+0000","last_fullsized":"2023-10-21T05:18:55.113580+0000","mapping_epoch":16,"log_start":"92'484625","ondisk_log_start":"92'484625","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":594,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 141s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6859608083,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":492053,"num_read_kb":1937244,"num_write":492629,"num_write_kb":1970510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"94'17082","reported_seq":23569,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:18:54.867939+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:18:54.867939+0000","last_peered":"2023-10-21T05:18:54.867939+0000","last_clean":"2023-10-21T05:18:54.867939+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:18:54.867939+0000","last_undegraded":"2023-10-21T05:18:54.867939+0000","last_fullsized":"2023-10-21T05:18:54.867939+0000","mapping_epoch":13,"log_start":"80'7000","ondisk_log_start":"80'7000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":269,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:59.707652+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:17:59.707652+0000","last_peered":"2023-10-21T05:17:59.707652+0000","last_clean":"2023-10-21T05:17:59.707652+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:17:59.707652+0000","last_undegraded":"2023-10-21T05:17:59.707652+0000","last_fullsized":"2023-10-21T05:17:59.707652+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27378135059,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1971243,"num_read_kb":7762352,"num_write":1969972,"num_write_kb":7879882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16138330112,"data_stored":19032301606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40128,"ondisk_log_size":40128,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37898,"ondisk_log_size":37898,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739300,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9805964,"kb_used_data":7880576,"kb_used_omap":787,"kb_used_meta":1924588,"kb_avail":84565876,"statfs":{"total":96636764160,"available":86595457024,"internally_reserved":0,"allocated":8069709824,"data_stored":9516162052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":806805,"internal_metadata":1970778219},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:00 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52300000000000002,"15min":0.497},"min":{"1min":0.19800000000000001,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":2.379,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.57199999999999995,"15min":0.497},"min":{"1min":0.192,"5min":0.192,"15min":0.185},"max":{"1min":2.3969999999999998,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.35599999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:18:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54600000000000004,"15min":0.58999999999999997},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.188},"max":{"1min":3.9910000000000001,"5min":5.2380000000000004,"15min":7.8449999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.54900000000000004,"15min":0.52900000000000003},"min":{"1min":0.223,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.52200000000000002,"5min":4.8979999999999997,"15min":5.2199999999999998},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739297,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5708440,"kb_used_data":3989712,"kb_used_omap":355,"kb_used_meta":1718364,"kb_avail":88663400,"statfs":{"total":96636764160,"available":90791321600,"internally_reserved":0,"allocated":4085465088,"data_stored":4831347798,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":1759605320},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,6,5,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:18:03 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.48899999999999999,"15min":0.51100000000000001},"min":{"1min":0.22700000000000001,"5min":0.19600000000000001,"15min":0.16900000000000001},"max":{"1min":6.173,"5min":6.173,"15min":6.173},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.47499999999999998,"15min":0.48099999999999998},"min":{"1min":0.27700000000000002,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:18:03 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.41099999999999998,"15min":0.45900000000000002},"min":{"1min":0.219,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.73,"5min":4.3259999999999996,"15min":8.7669999999999995},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.39900000000000002,"15min":0.40799999999999997},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":0.71399999999999997,"5min":4.1920000000000002,"15min":4.306},"last":0.32600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739301,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5538304,"kb_used_data":3891576,"kb_used_omap":331,"kb_used_meta":1646388,"kb_avail":88833536,"statfs":{"total":96636764160,"available":90965540864,"internally_reserved":0,"allocated":3984973824,"data_stored":4685844519,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":339590,"internal_metadata":1685901690},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:18:11 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.54900000000000004,"15min":0.42399999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":15.727,"5min":15.727,"15min":15.727},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.60599999999999998,"15min":0.47999999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.217},"max":{"1min":15.757999999999999,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.35999999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:18:11 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.51600000000000001,"15min":0.46800000000000003},"min":{"1min":0.221,"5min":0.221,"15min":0.22},"max":{"1min":0.49399999999999999,"5min":6.7430000000000003,"15min":6.7430000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.629,"15min":0.53900000000000003},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.23499999999999999},"max":{"1min":10.967000000000001,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3984187392,"data_stored":4685123584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4084817920,"data_stored":4830867475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8069324800,"data_stored":9516310547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:18:58.335 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:58.336 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:58.337 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:18:58.337 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-21T05:18:58.856 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-21T05:18:58.904 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-21T05:18:59.482 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-21T05:18:59.514 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:18:59.907 INFO:teuthology.orchestra.run.smithi133.stderr:2023-10-21T05:18:59.902+0000 7f6c8cbef640 0 --2- 172.21.15.133:0/618656807 >> v2:172.21.15.133:3300/0 conn(0x7f6c8812cf80 0x7f6c8812f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-21T05:19:00.181 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T05:19:00.181 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:19:00.182 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:19:00.182 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:19:00.182 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:19:00.182 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T05:19:00.183 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:19:00.183 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:19:00.183 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:19:00.184 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:19:00.184 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:19:00.184 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:19:00.184 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:19:00.185 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:00.174+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:19:00.274 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:19:00.315 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:19:10.266 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T05:19:10.267 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:19:10.267 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:19:10.267 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:19:10.268 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:19:10.268 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T05:19:10.268 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:19:10.269 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:19:10.269 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:19:10.269 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:19:10.269 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:19:10.270 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:19:10.270 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:19:10.270 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:10.262+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:19:15.600 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:15.590+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:19:15.605 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:15.598+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:19:15.610 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:15.606+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:19:20.316 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:19:20.808 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:19:20.808 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:19:20.840 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2428,"stamp":"2023-10-21T05:19:20.216408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27406056577,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":1997709,"num_read_kb":7808109,"num_write":1978705,"num_write_kb":7914063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78286,"ondisk_log_size":78286,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21300360,"kb_used_data":15829040,"kb_used_omap":1474,"kb_used_meta":5469821,"kb_avail":261815160,"statfs":{"total":289910292480,"available":268098723840,"internally_reserved":0,"allocated":16208936960,"data_stored":19102142593,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1510355,"internal_metadata":5601096749},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,3,9,6,11,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15409152,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4110,"num_read_kb":16404,"num_write":4493,"num_write_kb":17972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002397"},"pg_stats":[{"pgid":"2.7","version":"94'13757","reported_seq":18889,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:13.871097+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:19:13.871097+0000","last_peered":"2023-10-21T05:19:13.871097+0000","last_clean":"2023-10-21T05:19:13.871097+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:19:13.871097+0000","last_undegraded":"2023-10-21T05:19:13.871097+0000","last_fullsized":"2023-10-21T05:19:13.871097+0000","mapping_epoch":13,"log_start":"78'3700","ondisk_log_start":"78'3700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"94'7016","reported_seq":10624,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:13.855176+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:19:13.855176+0000","last_peered":"2023-10-21T05:19:13.855176+0000","last_clean":"2023-10-21T05:19:13.855176+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:19:13.855176+0000","last_undegraded":"2023-10-21T05:19:13.855176+0000","last_fullsized":"2023-10-21T05:19:13.855176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"94'3516","reported_seq":4963,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:12.968392+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:19:12.968392+0000","last_peered":"2023-10-21T05:19:12.968392+0000","last_clean":"2023-10-21T05:19:12.968392+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:19:12.968392+0000","last_undegraded":"2023-10-21T05:19:12.968392+0000","last_fullsized":"2023-10-21T05:19:12.968392+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"94'6962","reported_seq":9980,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:13.858298+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:19:13.858298+0000","last_peered":"2023-10-21T05:19:13.858298+0000","last_clean":"2023-10-21T05:19:13.858298+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:19:13.858298+0000","last_undegraded":"2023-10-21T05:19:13.858298+0000","last_fullsized":"2023-10-21T05:19:13.858298+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6962,"log_dups_size":0,"ondisk_log_size":6962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"94'506063","reported_seq":1006392,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:18.807152+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:19:18.807152+0000","last_peered":"2023-10-21T05:19:18.807152+0000","last_clean":"2023-10-21T05:19:18.807152+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:19:18.807152+0000","last_undegraded":"2023-10-21T05:19:18.807152+0000","last_fullsized":"2023-10-21T05:19:18.807152+0000","mapping_epoch":16,"log_start":"93'496018","ondisk_log_start":"93'496018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2645,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 698s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6942425088,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":501431,"num_read_kb":1974964,"num_write":502781,"num_write_kb":2011124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":224,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:18:39.962587+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:18:39.962587+0000","last_peered":"2023-10-21T05:18:39.962587+0000","last_clean":"2023-10-21T05:18:39.962587+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:18:39.962587+0000","last_undegraded":"2023-10-21T05:18:39.962587+0000","last_fullsized":"2023-10-21T05:18:39.962587+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"94'509895","reported_seq":1016483,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:15.261503+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:19:15.261503+0000","last_peered":"2023-10-21T05:19:15.261503+0000","last_clean":"2023-10-21T05:19:15.261503+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:19:15.261503+0000","last_undegraded":"2023-10-21T05:19:15.261503+0000","last_fullsized":"2023-10-21T05:19:15.261503+0000","mapping_epoch":16,"log_start":"92'499819","ondisk_log_start":"92'499819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2733,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 684s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7058874368,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":507765,"num_read_kb":1999672,"num_write":506458,"num_write_kb":2025832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4000,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:17.835870+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:19:17.835870+0000","last_peered":"2023-10-21T05:19:17.835870+0000","last_clean":"2023-10-21T05:19:17.835870+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:19:17.835870+0000","last_undegraded":"2023-10-21T05:19:17.835870+0000","last_fullsized":"2023-10-21T05:19:17.835870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"94'477574","reported_seq":952365,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:18.787781+0000","last_change":"2023-10-21T05:19:17.522440+0000","last_active":"2023-10-21T05:19:18.787781+0000","last_peered":"2023-10-21T05:19:18.787781+0000","last_clean":"2023-10-21T05:19:18.787781+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:19:18.787781+0000","last_undegraded":"2023-10-21T05:19:18.787781+0000","last_fullsized":"2023-10-21T05:19:18.787781+0000","mapping_epoch":16,"log_start":"92'467520","ondisk_log_start":"92'467520","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2779,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"queued for deep scrub","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6538489856,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":475747,"num_read_kb":1873436,"num_write":474332,"num_write_kb":1897328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3935,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:17.839064+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:19:17.839064+0000","last_peered":"2023-10-21T05:19:17.839064+0000","last_clean":"2023-10-21T05:19:17.839064+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:19:17.839064+0000","last_undegraded":"2023-10-21T05:19:17.839064+0000","last_fullsized":"2023-10-21T05:19:17.839064+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"94'496501","reported_seq":988967,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:15.315525+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:19:15.315525+0000","last_peered":"2023-10-21T05:19:15.315525+0000","last_clean":"2023-10-21T05:19:15.315525+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:19:15.315525+0000","last_undegraded":"2023-10-21T05:19:15.315525+0000","last_fullsized":"2023-10-21T05:19:15.315525+0000","mapping_epoch":16,"log_start":"92'486425","ondisk_log_start":"92'486425","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":665,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 161s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6865674259,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":493678,"num_read_kb":1943716,"num_write":494396,"num_write_kb":1977578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"94'17167","reported_seq":23675,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:13.939916+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:19:13.939916+0000","last_peered":"2023-10-21T05:19:13.939916+0000","last_clean":"2023-10-21T05:19:13.939916+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:19:13.939916+0000","last_undegraded":"2023-10-21T05:19:13.939916+0000","last_fullsized":"2023-10-21T05:19:13.939916+0000","mapping_epoch":13,"log_start":"80'7100","ondisk_log_start":"80'7100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":269,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:59.707652+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:17:59.707652+0000","last_peered":"2023-10-21T05:17:59.707652+0000","last_clean":"2023-10-21T05:17:59.707652+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:17:59.707652+0000","last_undegraded":"2023-10-21T05:17:59.707652+0000","last_fullsized":"2023-10-21T05:17:59.707652+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27405463571,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1978621,"num_read_kb":7791788,"num_write":1977967,"num_write_kb":7911862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16207441920,"data_stored":19101413414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40251,"ondisk_log_size":40251,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37936,"ondisk_log_size":37936,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739305,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9949628,"kb_used_data":7915956,"kb_used_omap":787,"kb_used_meta":2032876,"kb_avail":84422212,"statfs":{"total":96636764160,"available":86448345088,"internally_reserved":0,"allocated":8105938944,"data_stored":9552391172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":806805,"internal_metadata":2081665131},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.52900000000000003,"15min":0.52000000000000002},"min":{"1min":0.192,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":4.2619999999999996,"5min":4.7560000000000002,"15min":4.8540000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48399999999999999,"15min":0.504},"min":{"1min":0.254,"5min":0.192,"15min":0.185},"max":{"1min":1.5629999999999999,"5min":4.8449999999999998,"15min":8.8510000000000009},"last":0.27300000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.188},"max":{"1min":7.5099999999999998,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.31},{"interface":"front","average":{"1min":0.879,"5min":0.57999999999999996,"15min":0.56299999999999994},"min":{"1min":0.224,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":7.476,"5min":7.476,"15min":7.476},"last":0.29399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739301,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5754492,"kb_used_data":4003828,"kb_used_omap":355,"kb_used_meta":1750300,"kb_avail":88617348,"statfs":{"total":96636764160,"available":90744164352,"internally_reserved":0,"allocated":4099919872,"data_stored":4845802582,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":1792307784},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,7,2,6,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:03 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.46500000000000002,"15min":0.51200000000000001},"min":{"1min":0.28299999999999997,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.57499999999999996,"5min":6.173,"15min":6.173},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.46100000000000002,"15min":0.48199999999999998},"min":{"1min":0.26500000000000001,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":0.68200000000000005,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.26300000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:19:03 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.41999999999999998,"15min":0.46100000000000002},"min":{"1min":0.251,"5min":0.214,"15min":0.20399999999999999},"max":{"1min":0.79200000000000004,"5min":4.3259999999999996,"15min":8.7669999999999995},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40699999999999997,"15min":0.40799999999999997},"min":{"1min":0.25,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":0.52400000000000002,"5min":4.1920000000000002,"15min":4.306},"last":0.30599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739306,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5596240,"kb_used_data":3909256,"kb_used_omap":331,"kb_used_meta":1686644,"kb_avail":88775600,"statfs":{"total":96636764160,"available":90906214400,"internally_reserved":0,"allocated":4003078144,"data_stored":4703948839,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":339590,"internal_metadata":1727123834},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.54800000000000004,"15min":0.42499999999999999},"min":{"1min":0.246,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":0.58499999999999996,"5min":15.727,"15min":15.727},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.60299999999999998,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.247,"15min":0.217},"max":{"1min":0.65700000000000003,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.41499999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.46700000000000003,"15min":0.47699999999999998},"min":{"1min":0.27900000000000003,"5min":0.221,"15min":0.22},"max":{"1min":1.9359999999999999,"5min":4.2789999999999999,"15min":6.7430000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.622,"15min":0.54000000000000004},"min":{"1min":0.29699999999999999,"5min":0.252,"15min":0.23499999999999999},"max":{"1min":0.64200000000000002,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4002267136,"data_stored":4703203328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4099514368,"data_stored":4845563923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8105660416,"data_stored":9552646163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:19:20.842 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:20.842 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:20.843 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:20.843 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:19:27.349 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:27.342+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:19:27.366 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:27.350+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:19:27.366 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:27.350+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:19:27.370 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:27.366+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:19:27.371 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:27.366+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:19:31.283 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:31.278+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:19:31.283 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:19:31.278+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:19:40.846 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:19:41.390 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:19:41.391 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:19:41.433 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2438,"stamp":"2023-10-21T05:19:40.222802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27429739649,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2004709,"num_read_kb":7836041,"num_write":1985593,"num_write_kb":7941615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78201,"ondisk_log_size":78201,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20432516,"kb_used_data":15884008,"kb_used_omap":1627,"kb_used_meta":4546852,"kb_avail":262683004,"statfs":{"total":289910292480,"available":268987396096,"internally_reserved":0,"allocated":16265224192,"data_stored":19158429825,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1666269,"internal_metadata":4655977251},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,6,4,6,10,3,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14516224,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4417,"num_read_kb":17616,"num_write":4224,"num_write_kb":16896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002082"},"pg_stats":[{"pgid":"2.7","version":"94'13811","reported_seq":18959,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:35.295944+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:19:35.295944+0000","last_peered":"2023-10-21T05:19:35.295944+0000","last_clean":"2023-10-21T05:19:35.295944+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:19:35.295944+0000","last_undegraded":"2023-10-21T05:19:35.295944+0000","last_fullsized":"2023-10-21T05:19:35.295944+0000","mapping_epoch":13,"log_start":"80'3800","ondisk_log_start":"80'3800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"94'7045","reported_seq":10661,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:35.267732+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:19:35.267732+0000","last_peered":"2023-10-21T05:19:35.267732+0000","last_clean":"2023-10-21T05:19:35.267732+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:19:35.267732+0000","last_undegraded":"2023-10-21T05:19:35.267732+0000","last_fullsized":"2023-10-21T05:19:35.267732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"94'3531","reported_seq":4982,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:34.891055+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:19:34.891055+0000","last_peered":"2023-10-21T05:19:34.891055+0000","last_clean":"2023-10-21T05:19:34.891055+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:19:34.891055+0000","last_undegraded":"2023-10-21T05:19:34.891055+0000","last_fullsized":"2023-10-21T05:19:34.891055+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"94'6991","reported_seq":10017,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:35.266647+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:19:35.266647+0000","last_peered":"2023-10-21T05:19:35.266647+0000","last_clean":"2023-10-21T05:19:35.266647+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:19:35.266647+0000","last_undegraded":"2023-10-21T05:19:35.266647+0000","last_fullsized":"2023-10-21T05:19:35.266647+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":6991,"log_dups_size":0,"ondisk_log_size":6991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"94'507863","reported_seq":1009947,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:38.867015+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:19:38.867015+0000","last_peered":"2023-10-21T05:19:38.867015+0000","last_clean":"2023-10-21T05:19:38.867015+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:19:38.867015+0000","last_undegraded":"2023-10-21T05:19:38.867015+0000","last_fullsized":"2023-10-21T05:19:38.867015+0000","mapping_epoch":16,"log_start":"93'497818","ondisk_log_start":"93'497818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2699,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 719s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6948470784,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":503173,"num_read_kb":1981912,"num_write":504548,"num_write_kb":2018192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":224,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:18:39.962587+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:18:39.962587+0000","last_peered":"2023-10-21T05:18:39.962587+0000","last_clean":"2023-10-21T05:18:39.962587+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:18:39.962587+0000","last_undegraded":"2023-10-21T05:18:39.962587+0000","last_fullsized":"2023-10-21T05:18:39.962587+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"94'511733","reported_seq":1020109,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:35.446026+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:19:35.446026+0000","last_peered":"2023-10-21T05:19:35.446026+0000","last_clean":"2023-10-21T05:19:35.446026+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:19:35.446026+0000","last_undegraded":"2023-10-21T05:19:35.446026+0000","last_fullsized":"2023-10-21T05:19:35.446026+0000","mapping_epoch":16,"log_start":"93'501719","ondisk_log_start":"93'501719","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2797,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 704s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7065141248,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":509544,"num_read_kb":2006776,"num_write":508258,"num_write_kb":2033032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4012,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:38.312418+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:19:38.312418+0000","last_peered":"2023-10-21T05:19:38.312418+0000","last_clean":"2023-10-21T05:19:38.312418+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:19:38.312418+0000","last_undegraded":"2023-10-21T05:19:38.312418+0000","last_fullsized":"2023-10-21T05:19:38.312418+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"94'479208","reported_seq":955689,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:38.865915+0000","last_change":"2023-10-21T05:19:35.524107+0000","last_active":"2023-10-21T05:19:38.865915+0000","last_peered":"2023-10-21T05:19:38.865915+0000","last_clean":"2023-10-21T05:19:38.865915+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:19:38.865915+0000","last_undegraded":"2023-10-21T05:19:38.865915+0000","last_fullsized":"2023-10-21T05:19:38.865915+0000","mapping_epoch":16,"log_start":"93'469120","ondisk_log_start":"93'469120","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2779,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"queued for deep scrub","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6544015360,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":477428,"num_read_kb":1880144,"num_write":475966,"num_write_kb":1903864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3947,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:38.345579+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:19:38.345579+0000","last_peered":"2023-10-21T05:19:38.345579+0000","last_clean":"2023-10-21T05:19:38.345579+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:19:38.345579+0000","last_undegraded":"2023-10-21T05:19:38.345579+0000","last_fullsized":"2023-10-21T05:19:38.345579+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"94'498250","reported_seq":992529,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:35.451721+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:19:35.451721+0000","last_peered":"2023-10-21T05:19:35.451721+0000","last_clean":"2023-10-21T05:19:35.451721+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:19:35.451721+0000","last_undegraded":"2023-10-21T05:19:35.451721+0000","last_fullsized":"2023-10-21T05:19:35.451721+0000","mapping_epoch":16,"log_start":"93'488225","ondisk_log_start":"93'488225","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":761,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 181s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6871519251,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":495476,"num_read_kb":1950888,"num_write":496083,"num_write_kb":1984326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"94'17234","reported_seq":23762,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:35.319895+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:19:35.319895+0000","last_peered":"2023-10-21T05:19:35.319895+0000","last_clean":"2023-10-21T05:19:35.319895+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:19:35.319895+0000","last_undegraded":"2023-10-21T05:19:35.319895+0000","last_fullsized":"2023-10-21T05:19:35.319895+0000","mapping_epoch":13,"log_start":"80'7200","ondisk_log_start":"80'7200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":269,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-21T05:17:59.707652+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:17:59.707652+0000","last_peered":"2023-10-21T05:17:59.707652+0000","last_clean":"2023-10-21T05:17:59.707652+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:17:59.707652+0000","last_undegraded":"2023-10-21T05:17:59.707652+0000","last_fullsized":"2023-10-21T05:17:59.707652+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27429146643,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1985621,"num_read_kb":7819720,"num_write":1984855,"num_write_kb":7939414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16263716864,"data_stored":19157688358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40172,"ondisk_log_size":40172,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":37930,"ondisk_log_size":37930,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739309,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8942668,"kb_used_data":7943232,"kb_used_omap":940,"kb_used_meta":998483,"kb_avail":85429172,"statfs":{"total":96636764160,"available":87479472128,"internally_reserved":0,"allocated":8133869568,"data_stored":9580321796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":1022447457},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,3,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.52900000000000003,"15min":0.52000000000000002},"min":{"1min":0.192,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":4.2619999999999996,"5min":4.7560000000000002,"15min":4.8540000000000001},"last":3.9769999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48399999999999999,"15min":0.504},"min":{"1min":0.254,"5min":0.192,"15min":0.185},"max":{"1min":1.5629999999999999,"5min":4.8449999999999998,"15min":8.8510000000000009},"last":4.0830000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.188},"max":{"1min":7.5099999999999998,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.879,"5min":0.57999999999999996,"15min":0.56299999999999994},"min":{"1min":0.224,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":7.476,"5min":7.476,"15min":7.476},"last":0.221}]}]},{"osd":1,"up_from":8,"seq":34359739305,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5830024,"kb_used_data":4017920,"kb_used_omap":355,"kb_used_meta":1811740,"kb_avail":88541816,"statfs":{"total":96636764160,"available":90666819584,"internally_reserved":0,"allocated":4114350080,"data_stored":4860232790,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":1855222344},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,2,4,2,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:03 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.46500000000000002,"15min":0.51200000000000001},"min":{"1min":0.28299999999999997,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.57499999999999996,"5min":6.173,"15min":6.173},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.46100000000000002,"15min":0.48199999999999998},"min":{"1min":0.26500000000000001,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":0.68200000000000005,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.45400000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:19:03 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.41999999999999998,"15min":0.46100000000000002},"min":{"1min":0.251,"5min":0.214,"15min":0.20399999999999999},"max":{"1min":0.79200000000000004,"5min":4.3259999999999996,"15min":8.7669999999999995},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40699999999999997,"15min":0.40799999999999997},"min":{"1min":0.25,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":0.52400000000000002,"5min":4.1920000000000002,"15min":4.306},"last":0.48599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739310,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5659824,"kb_used_data":3922856,"kb_used_omap":331,"kb_used_meta":1736628,"kb_avail":88712016,"statfs":{"total":96636764160,"available":90841104384,"internally_reserved":0,"allocated":4017004544,"data_stored":4717875239,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":339590,"internal_metadata":1778307450},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.54800000000000004,"15min":0.42499999999999999},"min":{"1min":0.246,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":0.58499999999999996,"5min":15.727,"15min":15.727},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.60299999999999998,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.247,"15min":0.217},"max":{"1min":0.65700000000000003,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.37}]},{"osd":2,"last update":"Sat Oct 21 05:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.46700000000000003,"15min":0.47699999999999998},"min":{"1min":0.27900000000000003,"5min":0.221,"15min":0.22},"max":{"1min":1.9359999999999999,"5min":4.2789999999999999,"15min":6.7430000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.622,"15min":0.54000000000000004},"min":{"1min":0.29699999999999999,"5min":0.252,"15min":0.23499999999999999},"max":{"1min":0.64200000000000002,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4016189440,"data_stored":4717125632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4113784832,"data_stored":4859834387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8133742592,"data_stored":9580728339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:19:41.435 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:41.435 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:41.436 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:19:41.437 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:20:01.438 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:20:01.905 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:20:01.907 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:20:01.935 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2449,"stamp":"2023-10-21T05:20:00.226385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27454217345,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2011564,"num_read_kb":7863409,"num_write":1992664,"num_write_kb":7969899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78361,"ondisk_log_size":78361,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20654156,"kb_used_data":15940400,"kb_used_omap":1627,"kb_used_meta":4712100,"kb_avail":262461364,"statfs":{"total":289910292480,"available":268760436736,"internally_reserved":0,"allocated":16322969600,"data_stored":19216161385,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1666269,"internal_metadata":4825191203},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,4,5,4,10,10,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12304384,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3465,"num_read_kb":13836,"num_write":3554,"num_write_kb":14216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.223236"},"pg_stats":[{"pgid":"2.7","version":"95'13859","reported_seq":19024,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:19:55.584742+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:19:55.584742+0000","last_peered":"2023-10-21T05:19:55.584742+0000","last_clean":"2023-10-21T05:19:55.584742+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:19:55.584742+0000","last_undegraded":"2023-10-21T05:19:55.584742+0000","last_fullsized":"2023-10-21T05:19:55.584742+0000","mapping_epoch":13,"log_start":"80'3800","ondisk_log_start":"80'3800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'7069","reported_seq":10695,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:19:55.567361+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:19:55.567361+0000","last_peered":"2023-10-21T05:19:55.567361+0000","last_clean":"2023-10-21T05:19:55.567361+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:19:55.567361+0000","last_undegraded":"2023-10-21T05:19:55.567361+0000","last_fullsized":"2023-10-21T05:19:55.567361+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7069,"log_dups_size":0,"ondisk_log_size":7069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'3543","reported_seq":4999,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:19:53.672579+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:19:53.672579+0000","last_peered":"2023-10-21T05:19:53.672579+0000","last_clean":"2023-10-21T05:19:53.672579+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:19:53.672579+0000","last_undegraded":"2023-10-21T05:19:53.672579+0000","last_fullsized":"2023-10-21T05:19:53.672579+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3543,"log_dups_size":0,"ondisk_log_size":3543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'7015","reported_seq":10050,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:19:55.568203+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:19:55.568203+0000","last_peered":"2023-10-21T05:19:55.568203+0000","last_clean":"2023-10-21T05:19:55.568203+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:19:55.568203+0000","last_undegraded":"2023-10-21T05:19:55.568203+0000","last_fullsized":"2023-10-21T05:19:55.568203+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7015,"log_dups_size":0,"ondisk_log_size":7015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'509732","reported_seq":1013527,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:59.000473+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:19:59.000473+0000","last_peered":"2023-10-21T05:19:59.000473+0000","last_clean":"2023-10-21T05:19:59.000473+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:19:59.000473+0000","last_undegraded":"2023-10-21T05:19:59.000473+0000","last_fullsized":"2023-10-21T05:19:59.000473+0000","mapping_epoch":16,"log_start":"93'499718","ondisk_log_start":"93'499718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2773,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 739s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6954676224,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":504873,"num_read_kb":1988696,"num_write":506368,"num_write_kb":2025472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":225,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:08.686312+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:19:08.686312+0000","last_peered":"2023-10-21T05:19:08.686312+0000","last_clean":"2023-10-21T05:19:08.686312+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:19:08.686312+0000","last_undegraded":"2023-10-21T05:19:08.686312+0000","last_fullsized":"2023-10-21T05:19:08.686312+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'513552","reported_seq":1023689,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:55.570090+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:19:55.570090+0000","last_peered":"2023-10-21T05:19:55.570090+0000","last_clean":"2023-10-21T05:19:55.570090+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:19:55.570090+0000","last_undegraded":"2023-10-21T05:19:55.570090+0000","last_fullsized":"2023-10-21T05:19:55.570090+0000","mapping_epoch":16,"log_start":"93'503519","ondisk_log_start":"93'503519","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2851,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 724s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7071502336,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":511294,"num_read_kb":2013760,"num_write":510044,"num_write_kb":2040176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4025,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:19:58.792277+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:19:58.792277+0000","last_peered":"2023-10-21T05:19:58.792277+0000","last_clean":"2023-10-21T05:19:58.792277+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:19:58.792277+0000","last_undegraded":"2023-10-21T05:19:58.792277+0000","last_fullsized":"2023-10-21T05:19:58.792277+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'480884","reported_seq":959016,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:19:58.991221+0000","last_change":"2023-10-21T05:19:49.756747+0000","last_active":"2023-10-21T05:19:58.991221+0000","last_peered":"2023-10-21T05:19:58.991221+0000","last_clean":"2023-10-21T05:19:58.991221+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:19:58.991221+0000","last_undegraded":"2023-10-21T05:19:58.991221+0000","last_fullsized":"2023-10-21T05:19:58.991221+0000","mapping_epoch":16,"log_start":"93'470820","ondisk_log_start":"93'470820","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2779,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"queued for deep scrub","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6549684224,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":479075,"num_read_kb":1886724,"num_write":477642,"num_write_kb":1910568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3960,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:19:58.799828+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:19:58.799828+0000","last_peered":"2023-10-21T05:19:58.799828+0000","last_clean":"2023-10-21T05:19:58.799828+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:19:58.799828+0000","last_undegraded":"2023-10-21T05:19:58.799828+0000","last_fullsized":"2023-10-21T05:19:58.799828+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'500079","reported_seq":996127,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:19:55.585533+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:19:55.585533+0000","last_peered":"2023-10-21T05:19:55.585533+0000","last_clean":"2023-10-21T05:19:55.585533+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:19:55.585533+0000","last_undegraded":"2023-10-21T05:19:55.585533+0000","last_fullsized":"2023-10-21T05:19:55.585533+0000","mapping_epoch":16,"log_start":"93'490025","ondisk_log_start":"93'490025","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":824,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 202s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6877761555,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":497234,"num_read_kb":1957908,"num_write":497872,"num_write_kb":1991482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'17293","reported_seq":23839,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:19:53.673228+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:19:53.673228+0000","last_peered":"2023-10-21T05:19:53.673228+0000","last_clean":"2023-10-21T05:19:53.673228+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:19:53.673228+0000","last_undegraded":"2023-10-21T05:19:53.673228+0000","last_fullsized":"2023-10-21T05:19:53.673228+0000","mapping_epoch":13,"log_start":"80'7200","ondisk_log_start":"80'7200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":270,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:18:39.963932+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:18:39.963932+0000","last_peered":"2023-10-21T05:18:39.963932+0000","last_clean":"2023-10-21T05:18:39.963932+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:18:39.963932+0000","last_undegraded":"2023-10-21T05:18:39.963932+0000","last_fullsized":"2023-10-21T05:18:39.963932+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27453624339,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":1992476,"num_read_kb":7847088,"num_write":1991926,"num_write_kb":7967698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16321720320,"data_stored":19215691814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40165,"ondisk_log_size":40165,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38097,"ondisk_log_size":38097,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739313,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9063932,"kb_used_data":7971440,"kb_used_omap":940,"kb_used_meta":1091539,"kb_avail":85307908,"statfs":{"total":96636764160,"available":87355297792,"internally_reserved":0,"allocated":8162754560,"data_stored":9609202172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":1117736801},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.52900000000000003,"15min":0.52000000000000002},"min":{"1min":0.192,"5min":0.17299999999999999,"15min":0.153},"max":{"1min":4.2619999999999996,"5min":4.7560000000000002,"15min":4.8540000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48399999999999999,"15min":0.504},"min":{"1min":0.254,"5min":0.192,"15min":0.185},"max":{"1min":1.5629999999999999,"5min":4.8449999999999998,"15min":8.8510000000000009},"last":0.41199999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:19:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.188},"max":{"1min":7.5099999999999998,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.57999999999999996,"15min":0.56299999999999994},"min":{"1min":0.224,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":7.476,"5min":7.476,"15min":7.476},"last":0.26500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739309,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5880304,"kb_used_data":4032104,"kb_used_omap":355,"kb_used_meta":1847836,"kb_avail":88491536,"statfs":{"total":96636764160,"available":90615332864,"internally_reserved":0,"allocated":4128874496,"data_stored":4874752590,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":1892184648},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,4,0,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:19:03 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.46500000000000002,"15min":0.51200000000000001},"min":{"1min":0.28299999999999997,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.57499999999999996,"5min":6.173,"15min":6.173},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.46100000000000002,"15min":0.48199999999999998},"min":{"1min":0.26500000000000001,"5min":0.19800000000000001,"15min":0.186},"max":{"1min":0.68200000000000005,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.33700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:19:03 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.41999999999999998,"15min":0.46100000000000002},"min":{"1min":0.251,"5min":0.214,"15min":0.20399999999999999},"max":{"1min":0.79200000000000004,"5min":4.3259999999999996,"15min":8.7669999999999995},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40699999999999997,"15min":0.40799999999999997},"min":{"1min":0.25,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":0.52400000000000002,"5min":4.1920000000000002,"15min":4.306},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739314,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5709920,"kb_used_data":3936856,"kb_used_omap":331,"kb_used_meta":1772724,"kb_avail":88661920,"statfs":{"total":96636764160,"available":90789806080,"internally_reserved":0,"allocated":4031340544,"data_stored":4732206623,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":339590,"internal_metadata":1815269754},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,1,2,2,7,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.54800000000000004,"15min":0.42499999999999999},"min":{"1min":0.246,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":0.58499999999999996,"5min":15.727,"15min":15.727},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.60299999999999998,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.247,"15min":0.217},"max":{"1min":0.65700000000000003,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:19:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.46700000000000003,"15min":0.47699999999999998},"min":{"1min":0.27900000000000003,"5min":0.221,"15min":0.22},"max":{"1min":1.9359999999999999,"5min":4.2789999999999999,"15min":6.7430000000000003},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.622,"15min":0.54000000000000004},"min":{"1min":0.29699999999999999,"5min":0.252,"15min":0.23499999999999999},"max":{"1min":0.64200000000000002,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.44500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4030517248,"data_stored":4731453440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4128423936,"data_stored":4874473491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8162779136,"data_stored":9609764883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:20:01.938 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:01.938 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:01.939 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:01.939 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:20:21.941 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:20:22.436 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:20:22.437 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:20:22.471 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2460,"stamp":"2023-10-21T05:20:22.230188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27481590913,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2019583,"num_read_kb":7895433,"num_write":2000532,"num_write_kb":8001371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78468,"ondisk_log_size":78468,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20928012,"kb_used_data":15999156,"kb_used_omap":1627,"kb_used_meta":4927204,"kb_avail":262187508,"statfs":{"total":289910292480,"available":268480008192,"internally_reserved":0,"allocated":16383135744,"data_stored":19276327529,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1666269,"internal_metadata":5045457699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,2,9,11,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15384576,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4543,"num_read_kb":18148,"num_write":4447,"num_write_kb":17788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002134"},"pg_stats":[{"pgid":"2.7","version":"95'13943","reported_seq":19129,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:19.960140+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:20:19.960140+0000","last_peered":"2023-10-21T05:20:19.960140+0000","last_clean":"2023-10-21T05:20:19.960140+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:20:19.960140+0000","last_undegraded":"2023-10-21T05:20:19.960140+0000","last_fullsized":"2023-10-21T05:20:19.960140+0000","mapping_epoch":13,"log_start":"80'3900","ondisk_log_start":"80'3900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'7111","reported_seq":10747,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:19.943133+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:20:19.943133+0000","last_peered":"2023-10-21T05:20:19.943133+0000","last_clean":"2023-10-21T05:20:19.943133+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:20:19.943133+0000","last_undegraded":"2023-10-21T05:20:19.943133+0000","last_fullsized":"2023-10-21T05:20:19.943133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7111,"log_dups_size":0,"ondisk_log_size":7111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'3564","reported_seq":5025,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:19.529042+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:20:19.529042+0000","last_peered":"2023-10-21T05:20:19.529042+0000","last_clean":"2023-10-21T05:20:19.529042+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:20:19.529042+0000","last_undegraded":"2023-10-21T05:20:19.529042+0000","last_fullsized":"2023-10-21T05:20:19.529042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3564,"log_dups_size":0,"ondisk_log_size":3564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'7057","reported_seq":10102,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:19.936733+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:20:19.936733+0000","last_peered":"2023-10-21T05:20:19.936733+0000","last_clean":"2023-10-21T05:20:19.936733+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:20:19.936733+0000","last_undegraded":"2023-10-21T05:20:19.936733+0000","last_fullsized":"2023-10-21T05:20:19.936733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7057,"log_dups_size":0,"ondisk_log_size":7057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'511536","reported_seq":1017096,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:19.129892+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:20:19.129892+0000","last_peered":"2023-10-21T05:20:19.129892+0000","last_clean":"2023-10-21T05:20:19.129892+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:20:19.129892+0000","last_undegraded":"2023-10-21T05:20:19.129892+0000","last_fullsized":"2023-10-21T05:20:19.129892+0000","mapping_epoch":16,"log_start":"93'501518","ondisk_log_start":"93'501518","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2856,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 759s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6960603136,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":506630,"num_read_kb":1995716,"num_write":508120,"num_write_kb":2032480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":225,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:08.686312+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:19:08.686312+0000","last_peered":"2023-10-21T05:19:08.686312+0000","last_clean":"2023-10-21T05:19:08.686312+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:19:08.686312+0000","last_undegraded":"2023-10-21T05:19:08.686312+0000","last_fullsized":"2023-10-21T05:19:08.686312+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'515798","reported_seq":1028294,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:20.762128+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:20:20.762128+0000","last_peered":"2023-10-21T05:20:20.762128+0000","last_clean":"2023-10-21T05:20:20.762128+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:20:20.762128+0000","last_undegraded":"2023-10-21T05:20:20.762128+0000","last_fullsized":"2023-10-21T05:20:20.762128+0000","mapping_epoch":16,"log_start":"93'505719","ondisk_log_start":"93'505719","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2932,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 750s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7079387136,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":513642,"num_read_kb":2023140,"num_write":512241,"num_write_kb":2048964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4037,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:19.276262+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:20:19.276262+0000","last_peered":"2023-10-21T05:20:19.276262+0000","last_clean":"2023-10-21T05:20:19.276262+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:20:19.276262+0000","last_undegraded":"2023-10-21T05:20:19.276262+0000","last_fullsized":"2023-10-21T05:20:19.276262+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'482581","reported_seq":962401,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:19.123169+0000","last_change":"2023-10-21T05:20:18.747633+0000","last_active":"2023-10-21T05:20:19.123169+0000","last_peered":"2023-10-21T05:20:19.123169+0000","last_clean":"2023-10-21T05:20:19.123169+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:20:19.123169+0000","last_undegraded":"2023-10-21T05:20:19.123169+0000","last_fullsized":"2023-10-21T05:20:19.123169+0000","mapping_epoch":16,"log_start":"93'472520","ondisk_log_start":"93'472520","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2779,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"queued for deep scrub","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6555410432,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":480753,"num_read_kb":1893424,"num_write":479339,"num_write_kb":1917356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3973,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:19.281147+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:20:19.281147+0000","last_peered":"2023-10-21T05:20:19.281147+0000","last_clean":"2023-10-21T05:20:19.281147+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:20:19.281147+0000","last_undegraded":"2023-10-21T05:20:19.281147+0000","last_fullsized":"2023-10-21T05:20:19.281147+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'502345","reported_seq":1000635,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:20.794221+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:20:20.794221+0000","last_peered":"2023-10-21T05:20:20.794221+0000","last_clean":"2023-10-21T05:20:20.794221+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:20:20.794221+0000","last_undegraded":"2023-10-21T05:20:20.794221+0000","last_fullsized":"2023-10-21T05:20:20.794221+0000","mapping_epoch":16,"log_start":"93'492325","ondisk_log_start":"93'492325","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":896,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 227s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6885597203,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":499470,"num_read_kb":1966832,"num_write":500094,"num_write_kb":2000370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'17398","reported_seq":23972,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:20.020541+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:20:20.020541+0000","last_peered":"2023-10-21T05:20:20.020541+0000","last_clean":"2023-10-21T05:20:20.020541+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:20:20.020541+0000","last_undegraded":"2023-10-21T05:20:20.020541+0000","last_fullsized":"2023-10-21T05:20:20.020541+0000","mapping_epoch":13,"log_start":"80'7300","ondisk_log_start":"80'7300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":271,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.027032+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:19:50.027032+0000","last_peered":"2023-10-21T05:19:50.027032+0000","last_clean":"2023-10-21T05:19:50.027032+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:19:50.027032+0000","last_undegraded":"2023-10-21T05:19:50.027032+0000","last_fullsized":"2023-10-21T05:19:50.027032+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27480997907,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2000495,"num_read_kb":7879112,"num_write":1999794,"num_write_kb":7999170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16382517248,"data_stored":19276488742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40178,"ondisk_log_size":40178,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38191,"ondisk_log_size":38191,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739317,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9202360,"kb_used_data":7998896,"kb_used_omap":940,"kb_used_meta":1202515,"kb_avail":85169480,"statfs":{"total":96636764160,"available":87213547520,"internally_reserved":0,"allocated":8190869504,"data_stored":9637317116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":1231376225},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:06 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":0.71199999999999997,"15min":0.56999999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.216},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.62,"15min":0.53700000000000003},"min":{"1min":0.215,"5min":0.192,"15min":0.185},"max":{"1min":8.2439999999999998,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.22900000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:20:06 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.70699999999999996,"15min":0.628},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":4.3630000000000004,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.131,"5min":0.72899999999999998,"15min":0.60199999999999998},"min":{"1min":0.221,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":8.4030000000000005,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.35999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739314,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5952040,"kb_used_data":4049696,"kb_used_omap":355,"kb_used_meta":1901980,"kb_avail":88419800,"statfs":{"total":96636764160,"available":90541875200,"internally_reserved":0,"allocated":4146888704,"data_stored":4892766798,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":1947628104},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.45400000000000001,"15min":0.51100000000000001},"min":{"1min":0.20399999999999999,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.78200000000000003,"5min":6.173,"15min":6.173},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.46200000000000002,"15min":0.48199999999999998},"min":{"1min":0.26100000000000001,"5min":0.246,"15min":0.186},"max":{"1min":0.81999999999999995,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.40400000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.41199999999999998,"15min":0.45900000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":0.55200000000000005,"5min":4.3259999999999996,"15min":8.7669999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.39900000000000002,"15min":0.40699999999999997},"min":{"1min":0.255,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":0.48599999999999999,"5min":4.1920000000000002,"15min":4.306},"last":0.49199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739318,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5773612,"kb_used_data":3950564,"kb_used_omap":331,"kb_used_meta":1822708,"kb_avail":88598228,"statfs":{"total":96636764160,"available":90724585472,"internally_reserved":0,"allocated":4045377536,"data_stored":4746243615,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":339590,"internal_metadata":1866453370},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58699999999999997,"15min":0.439},"min":{"1min":0.23100000000000001,"5min":0.188,"15min":0.185},"max":{"1min":3.8180000000000001,"5min":15.727,"15min":15.727},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.64100000000000001,"15min":0.49399999999999999},"min":{"1min":0.32000000000000001,"5min":0.247,"15min":0.24399999999999999},"max":{"1min":3.859,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.22},"max":{"1min":3.831,"5min":4.2789999999999999,"15min":6.7430000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63200000000000001,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":5.1079999999999997,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4044636160,"data_stored":4745572352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4146520064,"data_stored":4892569619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8191361024,"data_stored":9638346771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:20:22.473 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:22.474 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:22.474 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:22.475 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:20:42.476 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:20:42.961 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:20:42.963 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:20:43.018 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2470,"stamp":"2023-10-21T05:20:42.233606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27505368193,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2026920,"num_read_kb":7924681,"num_write":2007413,"num_write_kb":8028895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78579,"ondisk_log_size":78579,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21241308,"kb_used_data":16055484,"kb_used_omap":1627,"kb_used_meta":5184164,"kb_avail":261874212,"statfs":{"total":289910292480,"available":268159193088,"internally_reserved":0,"allocated":16440815616,"data_stored":19334007401,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1666269,"internal_metadata":5308584739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,8,15,12,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14872576,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4674,"num_read_kb":18628,"num_write":4286,"num_write_kb":17144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002080"},"pg_stats":[{"pgid":"2.7","version":"95'14024","reported_seq":19226,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:40.119725+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:20:40.119725+0000","last_peered":"2023-10-21T05:20:40.119725+0000","last_clean":"2023-10-21T05:20:40.119725+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:20:40.119725+0000","last_undegraded":"2023-10-21T05:20:40.119725+0000","last_fullsized":"2023-10-21T05:20:40.119725+0000","mapping_epoch":13,"log_start":"80'4000","ondisk_log_start":"80'4000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'7152","reported_seq":10796,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:40.104393+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:20:40.104393+0000","last_peered":"2023-10-21T05:20:40.104393+0000","last_clean":"2023-10-21T05:20:40.104393+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:20:40.104393+0000","last_undegraded":"2023-10-21T05:20:40.104393+0000","last_fullsized":"2023-10-21T05:20:40.104393+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7152,"log_dups_size":0,"ondisk_log_size":7152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'3585","reported_seq":5050,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:39.752175+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:20:39.752175+0000","last_peered":"2023-10-21T05:20:39.752175+0000","last_clean":"2023-10-21T05:20:39.752175+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:20:39.752175+0000","last_undegraded":"2023-10-21T05:20:39.752175+0000","last_fullsized":"2023-10-21T05:20:39.752175+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'7098","reported_seq":10151,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:40.104763+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:20:40.104763+0000","last_peered":"2023-10-21T05:20:40.104763+0000","last_clean":"2023-10-21T05:20:40.104763+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:20:40.104763+0000","last_undegraded":"2023-10-21T05:20:40.104763+0000","last_fullsized":"2023-10-21T05:20:40.104763+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7098,"log_dups_size":0,"ondisk_log_size":7098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'513297","reported_seq":1020765,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:39.210952+0000","last_change":"2023-10-21T05:07:39.832435+0000","last_active":"2023-10-21T05:20:39.210952+0000","last_peered":"2023-10-21T05:20:39.210952+0000","last_clean":"2023-10-21T05:20:39.210952+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:20:39.210952+0000","last_undegraded":"2023-10-21T05:20:39.210952+0000","last_fullsized":"2023-10-21T05:20:39.210952+0000","mapping_epoch":16,"log_start":"94'503218","ondisk_log_start":"94'503218","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2953,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 779s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6966382592,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":508526,"num_read_kb":2003272,"num_write":509817,"num_write_kb":2039268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":225,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:08.686312+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:19:08.686312+0000","last_peered":"2023-10-21T05:19:08.686312+0000","last_clean":"2023-10-21T05:19:08.686312+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:19:08.686312+0000","last_undegraded":"2023-10-21T05:19:08.686312+0000","last_fullsized":"2023-10-21T05:19:08.686312+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'517581","reported_seq":1031936,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:40.881889+0000","last_change":"2023-10-21T05:07:50.645864+0000","last_active":"2023-10-21T05:20:40.881889+0000","last_peered":"2023-10-21T05:20:40.881889+0000","last_clean":"2023-10-21T05:20:40.881889+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:20:40.881889+0000","last_undegraded":"2023-10-21T05:20:40.881889+0000","last_fullsized":"2023-10-21T05:20:40.881889+0000","mapping_epoch":16,"log_start":"94'507519","ondisk_log_start":"94'507519","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2997,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 770s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7085555712,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":515490,"num_read_kb":2030500,"num_write":513982,"num_write_kb":2055928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4049,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:39.754336+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:20:39.754336+0000","last_peered":"2023-10-21T05:20:39.754336+0000","last_clean":"2023-10-21T05:20:39.754336+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:20:39.754336+0000","last_undegraded":"2023-10-21T05:20:39.754336+0000","last_fullsized":"2023-10-21T05:20:39.754336+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'484266","reported_seq":965878,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:39.214008+0000","last_change":"2023-10-21T05:20:37.894593+0000","last_active":"2023-10-21T05:20:39.214008+0000","last_peered":"2023-10-21T05:20:39.214008+0000","last_clean":"2023-10-21T05:20:39.214008+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:20:39.214008+0000","last_undegraded":"2023-10-21T05:20:39.214008+0000","last_fullsized":"2023-10-21T05:20:39.214008+0000","mapping_epoch":16,"log_start":"93'474220","ondisk_log_start":"93'474220","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2779,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"queued for deep scrub","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6561124352,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":482542,"num_read_kb":1900556,"num_write":481024,"num_write_kb":1924096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3985,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:39.764391+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:20:39.764391+0000","last_peered":"2023-10-21T05:20:39.764391+0000","last_clean":"2023-10-21T05:20:39.764391+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:20:39.764391+0000","last_undegraded":"2023-10-21T05:20:39.764391+0000","last_fullsized":"2023-10-21T05:20:39.764391+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'504142","reported_seq":1004245,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:40.883329+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:20:40.883329+0000","last_peered":"2023-10-21T05:20:40.883329+0000","last_clean":"2023-10-21T05:20:40.883329+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:20:40.883329+0000","last_undegraded":"2023-10-21T05:20:40.883329+0000","last_fullsized":"2023-10-21T05:20:40.883329+0000","mapping_epoch":16,"log_start":"94'494125","ondisk_log_start":"94'494125","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":960,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 247s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6891712531,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":501274,"num_read_kb":1974032,"num_write":501852,"num_write_kb":2007402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'17499","reported_seq":24093,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:40.172278+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:20:40.172278+0000","last_peered":"2023-10-21T05:20:40.172278+0000","last_clean":"2023-10-21T05:20:40.172278+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:20:40.172278+0000","last_undegraded":"2023-10-21T05:20:40.172278+0000","last_fullsized":"2023-10-21T05:20:40.172278+0000","mapping_epoch":13,"log_start":"80'7400","ondisk_log_start":"80'7400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":271,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:19:50.027032+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:19:50.027032+0000","last_peered":"2023-10-21T05:19:50.027032+0000","last_clean":"2023-10-21T05:19:50.027032+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:19:50.027032+0000","last_undegraded":"2023-10-21T05:19:50.027032+0000","last_fullsized":"2023-10-21T05:19:50.027032+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27504775187,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2007832,"num_read_kb":7908360,"num_write":2006675,"num_write_kb":8026694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16438759424,"data_stored":19332730918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40204,"ondisk_log_size":40204,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38276,"ondisk_log_size":38276,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739321,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9362956,"kb_used_data":8027456,"kb_used_omap":940,"kb_used_meta":1334547,"kb_avail":85008884,"statfs":{"total":96636764160,"available":87049097216,"internally_reserved":0,"allocated":8220114944,"data_stored":9666562556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":1366576993},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:06 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":0.71199999999999997,"15min":0.56999999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.62,"15min":0.53700000000000003},"min":{"1min":0.215,"5min":0.192,"15min":0.185},"max":{"1min":8.2439999999999998,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.27200000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:20:06 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.70699999999999996,"15min":0.628},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":4.3630000000000004,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.131,"5min":0.72899999999999998,"15min":0.60199999999999998},"min":{"1min":0.221,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":8.4030000000000005,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.437}]}]},{"osd":1,"up_from":8,"seq":34359739318,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6038412,"kb_used_data":4063872,"kb_used_omap":355,"kb_used_meta":1974172,"kb_avail":88333428,"statfs":{"total":96636764160,"available":90453430272,"internally_reserved":0,"allocated":4161404928,"data_stored":4907283022,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":363960,"internal_metadata":2021552712},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.45400000000000001,"15min":0.51100000000000001},"min":{"1min":0.20399999999999999,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.78200000000000003,"5min":6.173,"15min":6.173},"last":0.378},{"interface":"front","average":{"1min":0.371,"5min":0.46200000000000002,"15min":0.48199999999999998},"min":{"1min":0.26100000000000001,"5min":0.246,"15min":0.186},"max":{"1min":0.81999999999999995,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.31}]},{"osd":2,"last update":"Sat Oct 21 05:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.41199999999999998,"15min":0.45900000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":0.55200000000000005,"5min":4.3259999999999996,"15min":8.7669999999999995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.39900000000000002,"15min":0.40699999999999997},"min":{"1min":0.255,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":0.48599999999999999,"5min":4.1920000000000002,"15min":4.306},"last":0.443}]}]},{"osd":0,"up_from":8,"seq":34359739322,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5839940,"kb_used_data":3964156,"kb_used_omap":331,"kb_used_meta":1875444,"kb_avail":88531900,"statfs":{"total":96636764160,"available":90656665600,"internally_reserved":0,"allocated":4059295744,"data_stored":4760161823,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":339590,"internal_metadata":1920455034},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,7,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58699999999999997,"15min":0.439},"min":{"1min":0.23100000000000001,"5min":0.188,"15min":0.185},"max":{"1min":3.8180000000000001,"5min":15.727,"15min":15.727},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.64100000000000001,"15min":0.49399999999999999},"min":{"1min":0.32000000000000001,"5min":0.247,"15min":0.24399999999999999},"max":{"1min":3.859,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.434}]},{"osd":2,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.22},"max":{"1min":3.831,"5min":4.2789999999999999,"15min":6.7430000000000003},"last":0.312},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63200000000000001,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":5.1079999999999997,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.35499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4058484736,"data_stored":4759420928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4160851968,"data_stored":4906901523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8219422720,"data_stored":9666408467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:20:43.020 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:43.020 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:43.021 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:20:43.021 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:03.022 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:21:03.483 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:21:03.484 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:21:03.523 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2481,"stamp":"2023-10-21T05:21:02.237027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27529829505,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2033948,"num_read_kb":7952693,"num_write":2014457,"num_write_kb":8057071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78665,"ondisk_log_size":78665,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21014220,"kb_used_data":16111412,"kb_used_omap":1750,"kb_used_meta":4901033,"kb_avail":262101300,"statfs":{"total":289910292480,"available":268391731200,"internally_reserved":0,"allocated":16498085888,"data_stored":19391277673,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1792691,"internal_metadata":5018658125},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,11,10,10,16,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12341248,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3588,"num_read_kb":14316,"num_write":3574,"num_write_kb":14296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001689"},"pg_stats":[{"pgid":"2.7","version":"95'14064","reported_seq":19283,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.236593+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:21:00.236593+0000","last_peered":"2023-10-21T05:21:00.236593+0000","last_clean":"2023-10-21T05:21:00.236593+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:21:00.236593+0000","last_undegraded":"2023-10-21T05:21:00.236593+0000","last_fullsized":"2023-10-21T05:21:00.236593+0000","mapping_epoch":13,"log_start":"80'4000","ondisk_log_start":"80'4000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'7172","reported_seq":10825,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.236007+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:21:00.236007+0000","last_peered":"2023-10-21T05:21:00.236007+0000","last_clean":"2023-10-21T05:21:00.236007+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:21:00.236007+0000","last_undegraded":"2023-10-21T05:21:00.236007+0000","last_fullsized":"2023-10-21T05:21:00.236007+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7172,"log_dups_size":0,"ondisk_log_size":7172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"95'3595","reported_seq":5065,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.022467+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:21:00.022467+0000","last_peered":"2023-10-21T05:21:00.022467+0000","last_clean":"2023-10-21T05:21:00.022467+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:21:00.022467+0000","last_undegraded":"2023-10-21T05:21:00.022467+0000","last_fullsized":"2023-10-21T05:21:00.022467+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3595,"log_dups_size":0,"ondisk_log_size":3595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'7118","reported_seq":10180,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.235988+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:21:00.235988+0000","last_peered":"2023-10-21T05:21:00.235988+0000","last_clean":"2023-10-21T05:21:00.235988+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:21:00.235988+0000","last_undegraded":"2023-10-21T05:21:00.235988+0000","last_fullsized":"2023-10-21T05:21:00.235988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7118,"log_dups_size":0,"ondisk_log_size":7118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'515095","reported_seq":1024358,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:59.280638+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:20:59.280638+0000","last_peered":"2023-10-21T05:20:59.280638+0000","last_clean":"2023-10-21T05:20:59.280638+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:20:59.280638+0000","last_undegraded":"2023-10-21T05:20:59.280638+0000","last_fullsized":"2023-10-21T05:20:59.280638+0000","mapping_epoch":16,"log_start":"94'505018","ondisk_log_start":"94'505018","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6972616704,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":510312,"num_read_kb":2010392,"num_write":511604,"num_write_kb":2046416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":225,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-21T05:19:08.686312+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:19:08.686312+0000","last_peered":"2023-10-21T05:19:08.686312+0000","last_clean":"2023-10-21T05:19:08.686312+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:19:08.686312+0000","last_undegraded":"2023-10-21T05:19:08.686312+0000","last_fullsized":"2023-10-21T05:19:08.686312+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'519468","reported_seq":1035607,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.974207+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:21:00.974207+0000","last_peered":"2023-10-21T05:21:00.974207+0000","last_clean":"2023-10-21T05:21:00.974207+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:21:00.974207+0000","last_undegraded":"2023-10-21T05:21:00.974207+0000","last_fullsized":"2023-10-21T05:21:00.974207+0000","mapping_epoch":16,"log_start":"94'509419","ondisk_log_start":"94'509419","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7092117504,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":517259,"num_read_kb":2037560,"num_write":515832,"num_write_kb":2063328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4063,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.236508+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:21:00.236508+0000","last_peered":"2023-10-21T05:21:00.236508+0000","last_clean":"2023-10-21T05:21:00.236508+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:21:00.236508+0000","last_undegraded":"2023-10-21T05:21:00.236508+0000","last_fullsized":"2023-10-21T05:21:00.236508+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'485905","reported_seq":969201,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:20:59.278772+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:20:59.278772+0000","last_peered":"2023-10-21T05:20:59.278772+0000","last_clean":"2023-10-21T05:20:59.278772+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:20:59.278772+0000","last_undegraded":"2023-10-21T05:20:59.278772+0000","last_fullsized":"2023-10-21T05:20:59.278772+0000","mapping_epoch":16,"log_start":"94'475820","ondisk_log_start":"94'475820","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":104,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 16s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6566674432,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":484210,"num_read_kb":1907200,"num_write":482657,"num_write_kb":1930628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":3998,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.341242+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:21:00.341242+0000","last_peered":"2023-10-21T05:21:00.341242+0000","last_clean":"2023-10-21T05:21:00.341242+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:21:00.341242+0000","last_undegraded":"2023-10-21T05:21:00.341242+0000","last_fullsized":"2023-10-21T05:21:00.341242+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'505964","reported_seq":1007885,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:00.969473+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:21:00.969473+0000","last_peered":"2023-10-21T05:21:00.969473+0000","last_clean":"2023-10-21T05:21:00.969473+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:21:00.969473+0000","last_undegraded":"2023-10-21T05:21:00.969473+0000","last_fullsized":"2023-10-21T05:21:00.969473+0000","mapping_epoch":16,"log_start":"94'495925","ondisk_log_start":"94'495925","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1035,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 267s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6897827859,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":503079,"num_read_kb":1981220,"num_write":503626,"num_write_kb":2014498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'17549","reported_seq":24164,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.237291+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:21:00.237291+0000","last_peered":"2023-10-21T05:21:00.237291+0000","last_clean":"2023-10-21T05:21:00.237291+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:21:00.237291+0000","last_undegraded":"2023-10-21T05:21:00.237291+0000","last_fullsized":"2023-10-21T05:21:00.237291+0000","mapping_epoch":13,"log_start":"80'7500","ondisk_log_start":"80'7500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":272,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:00.633610+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:20:00.633610+0000","last_peered":"2023-10-21T05:20:00.633610+0000","last_clean":"2023-10-21T05:20:00.633610+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:20:00.633610+0000","last_undegraded":"2023-10-21T05:20:00.633610+0000","last_fullsized":"2023-10-21T05:20:00.633610+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27529236499,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2014860,"num_read_kb":7936372,"num_write":2013719,"num_write_kb":8054870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16496640000,"data_stored":19390611494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40250,"ondisk_log_size":40250,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38316,"ondisk_log_size":38316,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739325,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9472808,"kb_used_data":8055392,"kb_used_omap":940,"kb_used_meta":1416467,"kb_avail":84899032,"statfs":{"total":96636764160,"available":86936608768,"internally_reserved":0,"allocated":8248721408,"data_stored":9695169020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":1450463073},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:06 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":0.71199999999999997,"15min":0.56999999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.62,"15min":0.53700000000000003},"min":{"1min":0.215,"5min":0.192,"15min":0.185},"max":{"1min":8.2439999999999998,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.25700000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:20:06 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.70699999999999996,"15min":0.628},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":4.3630000000000004,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.131,"5min":0.72899999999999998,"15min":0.60199999999999998},"min":{"1min":0.221,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":8.4030000000000005,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.437}]}]},{"osd":1,"up_from":8,"seq":34359739322,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5619780,"kb_used_data":4078204,"kb_used_omap":478,"kb_used_meta":1541089,"kb_avail":88752060,"statfs":{"total":96636764160,"available":90882109440,"internally_reserved":0,"allocated":4176080896,"data_stored":4921958990,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1578075250},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,6,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.45400000000000001,"15min":0.51100000000000001},"min":{"1min":0.20399999999999999,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.78200000000000003,"5min":6.173,"15min":6.173},"last":0.183},{"interface":"front","average":{"1min":0.371,"5min":0.46200000000000002,"15min":0.48199999999999998},"min":{"1min":0.26100000000000001,"5min":0.246,"15min":0.186},"max":{"1min":0.81999999999999995,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.25900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:20:05 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.41199999999999998,"15min":0.45900000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":0.55200000000000005,"5min":4.3259999999999996,"15min":8.7669999999999995},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.39900000000000002,"15min":0.40699999999999997},"min":{"1min":0.255,"5min":0.23000000000000001,"15min":0.20499999999999999},"max":{"1min":0.48599999999999999,"5min":4.1920000000000002,"15min":4.306},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739326,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5921632,"kb_used_data":3977816,"kb_used_omap":331,"kb_used_meta":1943476,"kb_avail":88450208,"statfs":{"total":96636764160,"available":90573012992,"internally_reserved":0,"allocated":4073283584,"data_stored":4774149663,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":339590,"internal_metadata":1990119802},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,2,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58699999999999997,"15min":0.439},"min":{"1min":0.23100000000000001,"5min":0.188,"15min":0.185},"max":{"1min":3.8180000000000001,"5min":15.727,"15min":15.727},"last":0.499},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.64100000000000001,"15min":0.49399999999999999},"min":{"1min":0.32000000000000001,"5min":0.247,"15min":0.24399999999999999},"max":{"1min":3.859,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.22},"max":{"1min":3.831,"5min":4.2789999999999999,"15min":6.7430000000000003},"last":0.376},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63200000000000001,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":5.1079999999999997,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4072497152,"data_stored":4773433344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4175704064,"data_stored":4921753619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8248438784,"data_stored":9695424531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:21:03.525 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:03.525 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:08.782 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:08.774+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:21:08.783 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:08.774+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:21:08.783 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:08.774+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:21:08.783 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:08.774+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:21:08.792 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:08.786+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:21:08.797 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:08.790+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:21:08.809 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:08.802+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:21:08.809 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:08.802+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:21:23.527 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:21:24.767 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:21:24.768 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:21:24.822 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2492,"stamp":"2023-10-21T05:21:24.240828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27555191937,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2041592,"num_read_kb":7983133,"num_write":2021751,"num_write_kb":8086247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78622,"ondisk_log_size":78622,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20545756,"kb_used_data":16170116,"kb_used_omap":1923,"kb_used_meta":4373692,"kb_avail":262569764,"statfs":{"total":289910292480,"available":268871438336,"internally_reserved":0,"allocated":16558198784,"data_stored":19451376592,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1969844,"internal_metadata":4478660940},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,3,15,14,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12787712,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3876,"num_read_kb":15416,"num_write":3681,"num_write_kb":14724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002138"},"pg_stats":[{"pgid":"2.7","version":"96'14118","reported_seq":19353,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:20.738693+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:21:20.738693+0000","last_peered":"2023-10-21T05:21:20.738693+0000","last_clean":"2023-10-21T05:21:20.738693+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:21:20.738693+0000","last_undegraded":"2023-10-21T05:21:20.738693+0000","last_fullsized":"2023-10-21T05:21:20.738693+0000","mapping_epoch":13,"log_start":"80'4100","ondisk_log_start":"80'4100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7200","reported_seq":10861,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:20.716765+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:21:20.716765+0000","last_peered":"2023-10-21T05:21:20.716765+0000","last_clean":"2023-10-21T05:21:20.716765+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:21:20.716765+0000","last_undegraded":"2023-10-21T05:21:20.716765+0000","last_fullsized":"2023-10-21T05:21:20.716765+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7200,"log_dups_size":0,"ondisk_log_size":7200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3609","reported_seq":5084,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:20.711433+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:21:20.711433+0000","last_peered":"2023-10-21T05:21:20.711433+0000","last_clean":"2023-10-21T05:21:20.711433+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:21:20.711433+0000","last_undegraded":"2023-10-21T05:21:20.711433+0000","last_fullsized":"2023-10-21T05:21:20.711433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7146","reported_seq":10217,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:20.717341+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:21:20.717341+0000","last_peered":"2023-10-21T05:21:20.717341+0000","last_clean":"2023-10-21T05:21:20.717341+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:21:20.717341+0000","last_undegraded":"2023-10-21T05:21:20.717341+0000","last_fullsized":"2023-10-21T05:21:20.717341+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7146,"log_dups_size":0,"ondisk_log_size":7146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'516992","reported_seq":1028185,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:19.386967+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:21:19.386967+0000","last_peered":"2023-10-21T05:21:19.386967+0000","last_clean":"2023-10-21T05:21:19.386967+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:21:19.386967+0000","last_undegraded":"2023-10-21T05:21:19.386967+0000","last_fullsized":"2023-10-21T05:21:19.386967+0000","mapping_epoch":16,"log_start":"94'506918","ondisk_log_start":"94'506918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6979158016,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":512246,"num_read_kb":2018104,"num_write":513501,"num_write_kb":2054004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":227,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.026502+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:21:00.026502+0000","last_peered":"2023-10-21T05:21:00.026502+0000","last_clean":"2023-10-21T05:21:00.026502+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:21:00.026502+0000","last_undegraded":"2023-10-21T05:21:00.026502+0000","last_fullsized":"2023-10-21T05:21:00.026502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'521319","reported_seq":1039467,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:21.092893+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:21:21.092893+0000","last_peered":"2023-10-21T05:21:21.092893+0000","last_clean":"2023-10-21T05:21:21.092893+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:21:21.092893+0000","last_undegraded":"2023-10-21T05:21:21.092893+0000","last_fullsized":"2023-10-21T05:21:21.092893+0000","mapping_epoch":16,"log_start":"94'511219","ondisk_log_start":"94'511219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7098667008,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":519266,"num_read_kb":2045544,"num_write":517683,"num_write_kb":2070732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4072,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:15.593936+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:21:15.593936+0000","last_peered":"2023-10-21T05:21:15.593936+0000","last_clean":"2023-10-21T05:21:15.593936+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:21:15.593936+0000","last_undegraded":"2023-10-21T05:21:15.593936+0000","last_fullsized":"2023-10-21T05:21:15.593936+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'487635","reported_seq":972748,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:19.364506+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:21:19.364506+0000","last_peered":"2023-10-21T05:21:19.364506+0000","last_clean":"2023-10-21T05:21:19.364506+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:21:19.364506+0000","last_undegraded":"2023-10-21T05:21:19.364506+0000","last_fullsized":"2023-10-21T05:21:19.364506+0000","mapping_epoch":16,"log_start":"94'477620","ondisk_log_start":"94'477620","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":233,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 36s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6572568576,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":486015,"num_read_kb":1914388,"num_write":484381,"num_write_kb":1937524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4007,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:15.593351+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:21:15.593351+0000","last_peered":"2023-10-21T05:21:15.593351+0000","last_clean":"2023-10-21T05:21:15.593351+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:21:15.593351+0000","last_undegraded":"2023-10-21T05:21:15.593351+0000","last_fullsized":"2023-10-21T05:21:15.593351+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'507851","reported_seq":1011680,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:21.107512+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:21:21.107512+0000","last_peered":"2023-10-21T05:21:21.107512+0000","last_clean":"2023-10-21T05:21:21.107512+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:21:21.107512+0000","last_undegraded":"2023-10-21T05:21:21.107512+0000","last_fullsized":"2023-10-21T05:21:21.107512+0000","mapping_epoch":16,"log_start":"94'497825","ondisk_log_start":"94'497825","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1138,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 287s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6904205331,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":504977,"num_read_kb":1988776,"num_write":505448,"num_write_kb":2021786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'17617","reported_seq":24253,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:20.813755+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:21:20.813755+0000","last_peered":"2023-10-21T05:21:20.813755+0000","last_clean":"2023-10-21T05:21:20.813755+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:21:20.813755+0000","last_undegraded":"2023-10-21T05:21:20.813755+0000","last_fullsized":"2023-10-21T05:21:20.813755+0000","mapping_epoch":13,"log_start":"80'7600","ondisk_log_start":"80'7600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":272,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:00.633610+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:20:00.633610+0000","last_peered":"2023-10-21T05:20:00.633610+0000","last_clean":"2023-10-21T05:20:00.633610+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:20:00.633610+0000","last_undegraded":"2023-10-21T05:20:00.633610+0000","last_fullsized":"2023-10-21T05:20:00.633610+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27554598931,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2022504,"num_read_kb":7966812,"num_write":2021013,"num_write_kb":8084046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16556552192,"data_stored":19450523686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40215,"ondisk_log_size":40215,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38308,"ondisk_log_size":38308,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739329,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9605864,"kb_used_data":8084320,"kb_used_omap":940,"kb_used_meta":1520595,"kb_avail":84765976,"statfs":{"total":96636764160,"available":86800359424,"internally_reserved":0,"allocated":8278343680,"data_stored":9724786633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":1557090145},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.73199999999999998,"15min":0.57399999999999995},"min":{"1min":0.17499999999999999,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":1.849,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.63200000000000001,"15min":0.53800000000000003},"min":{"1min":0.22700000000000001,"5min":0.192,"15min":0.185},"max":{"1min":1.8080000000000001,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.501}]},{"osd":1,"last update":"Sat Oct 21 05:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.61899999999999999,"15min":0.59399999999999997},"min":{"1min":0.20599999999999999,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":0.68899999999999995,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.71299999999999997,"15min":0.57899999999999996},"min":{"1min":0.245,"5min":0.221,"15min":0.19800000000000001},"max":{"1min":0.68600000000000005,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.28399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739327,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5689264,"kb_used_data":4093544,"kb_used_omap":478,"kb_used_meta":1595233,"kb_avail":88682576,"statfs":{"total":96636764160,"available":90810957824,"internally_reserved":0,"allocated":4191789056,"data_stored":4937662491,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1633518706},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,7,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.57699999999999996,"15min":0.53500000000000003},"min":{"1min":0.183,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.59899999999999998,"15min":0.51700000000000002},"min":{"1min":0.254,"5min":0.249,"15min":0.186},"max":{"1min":6.7110000000000003,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.24399999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:21:06 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.378,"15min":0.44500000000000001},"min":{"1min":0.21299999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":0.57699999999999996,"5min":1.73,"15min":8.7669999999999995},"last":0.37},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.375,"15min":0.40699999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":1.468,"5min":1.468,"15min":4.306},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739330,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5250628,"kb_used_data":3992252,"kb_used_omap":504,"kb_used_meta":1257863,"kb_avail":89121212,"statfs":{"total":96636764160,"available":91260121088,"internally_reserved":0,"allocated":4088066048,"data_stored":4788927468,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1288052089},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58699999999999997,"15min":0.439},"min":{"1min":0.23100000000000001,"5min":0.188,"15min":0.185},"max":{"1min":3.8180000000000001,"5min":15.727,"15min":15.727},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.64100000000000001,"15min":0.49399999999999999},"min":{"1min":0.32000000000000001,"5min":0.247,"15min":0.24399999999999999},"max":{"1min":3.859,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:20:17 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.22},"max":{"1min":3.831,"5min":4.2789999999999999,"15min":6.7430000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63200000000000001,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":5.1079999999999997,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.40400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4087324672,"data_stored":4788260864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4190740480,"data_stored":4936790035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8278487040,"data_stored":9725472787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:21:24.824 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:24.825 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:21:27.114 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:27.110+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:21:27.119 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:27.114+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:21:27.119 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:27.114+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:21:27.122 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:27.114+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:21:27.130 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:27.126+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:21:27.131 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:27.126+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:21:27.132 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:21:27.126+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:21:44.826 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:21:45.357 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:21:45.359 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:21:45.396 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2502,"stamp":"2023-10-21T05:21:44.244179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27580603521,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2049147,"num_read_kb":8013213,"num_write":2029066,"num_write_kb":8115507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78733,"ondisk_log_size":78733,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20796032,"kb_used_data":16228648,"kb_used_omap":1923,"kb_used_meta":4565436,"kb_avail":262319488,"statfs":{"total":289910292480,"available":268615155712,"internally_reserved":0,"allocated":16618135552,"data_stored":19511313360,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1969844,"internal_metadata":4675006796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,2,11,7,14,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12943360,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3779,"num_read_kb":15048,"num_write":3700,"num_write_kb":14800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001975"},"pg_stats":[{"pgid":"2.7","version":"96'14188","reported_seq":19439,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:41.192445+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:21:41.192445+0000","last_peered":"2023-10-21T05:21:41.192445+0000","last_clean":"2023-10-21T05:21:41.192445+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:21:41.192445+0000","last_undegraded":"2023-10-21T05:21:41.192445+0000","last_fullsized":"2023-10-21T05:21:41.192445+0000","mapping_epoch":13,"log_start":"80'4100","ondisk_log_start":"80'4100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7236","reported_seq":10905,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:41.191959+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:21:41.191959+0000","last_peered":"2023-10-21T05:21:41.191959+0000","last_clean":"2023-10-21T05:21:41.191959+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:21:41.191959+0000","last_undegraded":"2023-10-21T05:21:41.191959+0000","last_fullsized":"2023-10-21T05:21:41.191959+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7236,"log_dups_size":0,"ondisk_log_size":7236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3627","reported_seq":5106,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:39.943671+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:21:39.943671+0000","last_peered":"2023-10-21T05:21:39.943671+0000","last_clean":"2023-10-21T05:21:39.943671+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:21:39.943671+0000","last_undegraded":"2023-10-21T05:21:39.943671+0000","last_fullsized":"2023-10-21T05:21:39.943671+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7182","reported_seq":10261,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:41.192011+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:21:41.192011+0000","last_peered":"2023-10-21T05:21:41.192011+0000","last_clean":"2023-10-21T05:21:41.192011+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:21:41.192011+0000","last_undegraded":"2023-10-21T05:21:41.192011+0000","last_fullsized":"2023-10-21T05:21:41.192011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7182,"log_dups_size":0,"ondisk_log_size":7182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'518833","reported_seq":1031950,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:39.478055+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:21:39.478055+0000","last_peered":"2023-10-21T05:21:39.478055+0000","last_clean":"2023-10-21T05:21:39.478055+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:21:39.478055+0000","last_undegraded":"2023-10-21T05:21:39.478055+0000","last_fullsized":"2023-10-21T05:21:39.478055+0000","mapping_epoch":16,"log_start":"94'508818","ondisk_log_start":"94'508818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6985478144,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":514167,"num_read_kb":2025736,"num_write":515342,"num_write_kb":2061368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":227,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.026502+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:21:00.026502+0000","last_peered":"2023-10-21T05:21:00.026502+0000","last_clean":"2023-10-21T05:21:00.026502+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:21:00.026502+0000","last_undegraded":"2023-10-21T05:21:00.026502+0000","last_fullsized":"2023-10-21T05:21:00.026502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'523193","reported_seq":1043216,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:41.224159+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:21:41.224159+0000","last_peered":"2023-10-21T05:21:41.224159+0000","last_clean":"2023-10-21T05:21:41.224159+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:21:41.224159+0000","last_undegraded":"2023-10-21T05:21:41.224159+0000","last_fullsized":"2023-10-21T05:21:41.224159+0000","mapping_epoch":16,"log_start":"95'513119","ondisk_log_start":"95'513119","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7105384448,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":521144,"num_read_kb":2053024,"num_write":519557,"num_write_kb":2078228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4084,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:36.078548+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:21:36.078548+0000","last_peered":"2023-10-21T05:21:36.078548+0000","last_clean":"2023-10-21T05:21:36.078548+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:21:36.078548+0000","last_undegraded":"2023-10-21T05:21:36.078548+0000","last_fullsized":"2023-10-21T05:21:36.078548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'489435","reported_seq":976428,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:39.481394+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:21:39.481394+0000","last_peered":"2023-10-21T05:21:39.481394+0000","last_clean":"2023-10-21T05:21:39.481394+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:21:39.481394+0000","last_undegraded":"2023-10-21T05:21:39.481394+0000","last_fullsized":"2023-10-21T05:21:39.481394+0000","mapping_epoch":16,"log_start":"94'479420","ondisk_log_start":"94'479420","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":320,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 56s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6578622464,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":487883,"num_read_kb":1921844,"num_write":486175,"num_write_kb":1944700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4019,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:36.096510+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:21:36.096510+0000","last_peered":"2023-10-21T05:21:36.096510+0000","last_clean":"2023-10-21T05:21:36.096510+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:21:36.096510+0000","last_undegraded":"2023-10-21T05:21:36.096510+0000","last_fullsized":"2023-10-21T05:21:36.096510+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'509700","reported_seq":1015434,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:21:41.239773+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:21:41.239773+0000","last_peered":"2023-10-21T05:21:41.239773+0000","last_clean":"2023-10-21T05:21:41.239773+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:21:41.239773+0000","last_undegraded":"2023-10-21T05:21:41.239773+0000","last_fullsized":"2023-10-21T05:21:41.239773+0000","mapping_epoch":16,"log_start":"95'499625","ondisk_log_start":"95'499625","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1210,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 307s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6910525459,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":506865,"num_read_kb":1996288,"num_write":507254,"num_write_kb":2029010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'17704","reported_seq":24360,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:41.192583+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:21:41.192583+0000","last_peered":"2023-10-21T05:21:41.192583+0000","last_clean":"2023-10-21T05:21:41.192583+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:21:41.192583+0000","last_undegraded":"2023-10-21T05:21:41.192583+0000","last_fullsized":"2023-10-21T05:21:41.192583+0000","mapping_epoch":13,"log_start":"80'7700","ondisk_log_start":"80'7700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":272,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-21T05:20:00.633610+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:20:00.633610+0000","last_peered":"2023-10-21T05:20:00.633610+0000","last_clean":"2023-10-21T05:20:00.633610+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:20:00.633610+0000","last_undegraded":"2023-10-21T05:20:00.633610+0000","last_fullsized":"2023-10-21T05:20:00.633610+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27580010515,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2030059,"num_read_kb":7996892,"num_write":2028328,"num_write_kb":8113306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16616321024,"data_stored":19510292518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40179,"ondisk_log_size":40179,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38455,"ondisk_log_size":38455,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739333,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9738592,"kb_used_data":8114328,"kb_used_omap":940,"kb_used_meta":1623315,"kb_avail":84633248,"statfs":{"total":96636764160,"available":86664445952,"internally_reserved":0,"allocated":8309071872,"data_stored":9755514825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":1662275425},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.73199999999999998,"15min":0.57399999999999995},"min":{"1min":0.17499999999999999,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":1.849,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.63200000000000001,"15min":0.53800000000000003},"min":{"1min":0.22700000000000001,"5min":0.192,"15min":0.185},"max":{"1min":1.8080000000000001,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.19700000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.61899999999999999,"15min":0.59399999999999997},"min":{"1min":0.20599999999999999,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":0.68899999999999995,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.71299999999999997,"15min":0.57899999999999996},"min":{"1min":0.245,"5min":0.221,"15min":0.19800000000000001},"max":{"1min":0.68600000000000005,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.28999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739331,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5753408,"kb_used_data":4107704,"kb_used_omap":478,"kb_used_meta":1645217,"kb_avail":88618432,"statfs":{"total":96636764160,"available":90745274368,"internally_reserved":0,"allocated":4206288896,"data_stored":4952162331,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1684702322},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,7,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.57699999999999996,"15min":0.53500000000000003},"min":{"1min":0.183,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.59899999999999998,"15min":0.51700000000000002},"min":{"1min":0.254,"5min":0.249,"15min":0.186},"max":{"1min":6.7110000000000003,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.255}]},{"osd":2,"last update":"Sat Oct 21 05:21:06 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.378,"15min":0.44500000000000001},"min":{"1min":0.21299999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":0.57699999999999996,"5min":1.73,"15min":8.7669999999999995},"last":0.312},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.375,"15min":0.40699999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":1.468,"5min":1.468,"15min":4.306},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739334,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5304032,"kb_used_data":4006616,"kb_used_omap":504,"kb_used_meta":1296903,"kb_avail":89067808,"statfs":{"total":96636764160,"available":91205435392,"internally_reserved":0,"allocated":4102774784,"data_stored":4803636204,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1328029049},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,1,5,8,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:21:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55400000000000005,"15min":0.45000000000000001},"min":{"1min":0.25800000000000001,"5min":0.188,"15min":0.185},"max":{"1min":2.5390000000000001,"5min":15.727,"15min":15.727},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60399999999999998,"15min":0.505},"min":{"1min":0.29099999999999998,"5min":0.247,"15min":0.24399999999999999},"max":{"1min":2.7850000000000001,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:21:20 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.46300000000000002,"15min":0.48899999999999999},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.22},"max":{"1min":0.80300000000000005,"5min":3.831,"15min":6.7430000000000003},"last":0.22},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.61699999999999999,"15min":0.56899999999999995},"min":{"1min":0.27800000000000002,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":2.8450000000000002,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4102213632,"data_stored":4803149824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4205830144,"data_stored":4951879699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8308277248,"data_stored":9755262995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:21:45.398 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:21:45.399 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:22:05.400 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:22:06.409 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:22:06.409 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:22:06.436 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2513,"stamp":"2023-10-21T05:22:06.248008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27608886401,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2057380,"num_read_kb":8045989,"num_write":2037305,"num_write_kb":8148463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78848,"ondisk_log_size":78848,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21143092,"kb_used_data":16297948,"kb_used_omap":1923,"kb_used_meta":4843196,"kb_avail":261972428,"statfs":{"total":289910292480,"available":268259766272,"internally_reserved":0,"allocated":16689098752,"data_stored":19582276560,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1969844,"internal_metadata":4959433036},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,6,15,19,23],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15724544,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4665,"num_read_kb":18556,"num_write":4581,"num_write_kb":18324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002018"},"pg_stats":[{"pgid":"2.7","version":"96'14232","reported_seq":19495,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.327093+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:22:01.327093+0000","last_peered":"2023-10-21T05:22:01.327093+0000","last_clean":"2023-10-21T05:22:01.327093+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:22:01.327093+0000","last_undegraded":"2023-10-21T05:22:01.327093+0000","last_fullsized":"2023-10-21T05:22:01.327093+0000","mapping_epoch":13,"log_start":"80'4200","ondisk_log_start":"80'4200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7258","reported_seq":10933,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.316436+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:22:01.316436+0000","last_peered":"2023-10-21T05:22:01.316436+0000","last_clean":"2023-10-21T05:22:01.316436+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:22:01.316436+0000","last_undegraded":"2023-10-21T05:22:01.316436+0000","last_fullsized":"2023-10-21T05:22:01.316436+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7258,"log_dups_size":0,"ondisk_log_size":7258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3638","reported_seq":5120,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:00.964234+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:22:00.964234+0000","last_peered":"2023-10-21T05:22:00.964234+0000","last_clean":"2023-10-21T05:22:00.964234+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:22:00.964234+0000","last_undegraded":"2023-10-21T05:22:00.964234+0000","last_fullsized":"2023-10-21T05:22:00.964234+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3638,"log_dups_size":0,"ondisk_log_size":3638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7204","reported_seq":10289,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.319716+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:22:01.319716+0000","last_peered":"2023-10-21T05:22:01.319716+0000","last_clean":"2023-10-21T05:22:01.319716+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:22:01.319716+0000","last_undegraded":"2023-10-21T05:22:01.319716+0000","last_fullsized":"2023-10-21T05:22:01.319716+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7204,"log_dups_size":0,"ondisk_log_size":7204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'521170","reported_seq":1036597,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:04.573340+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:22:04.573340+0000","last_peered":"2023-10-21T05:22:04.573340+0000","last_clean":"2023-10-21T05:22:04.573340+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:22:04.573340+0000","last_undegraded":"2023-10-21T05:22:04.573340+0000","last_fullsized":"2023-10-21T05:22:04.573340+0000","mapping_epoch":16,"log_start":"95'511118","ondisk_log_start":"95'511118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6993420288,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":516475,"num_read_kb":2034940,"num_write":517679,"num_write_kb":2070716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":227,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.026502+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:21:00.026502+0000","last_peered":"2023-10-21T05:21:00.026502+0000","last_clean":"2023-10-21T05:21:00.026502+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:21:00.026502+0000","last_undegraded":"2023-10-21T05:21:00.026502+0000","last_fullsized":"2023-10-21T05:21:00.026502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'525059","reported_seq":1046962,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.364180+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:22:01.364180+0000","last_peered":"2023-10-21T05:22:01.364180+0000","last_clean":"2023-10-21T05:22:01.364180+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:22:01.364180+0000","last_undegraded":"2023-10-21T05:22:01.364180+0000","last_fullsized":"2023-10-21T05:22:01.364180+0000","mapping_epoch":16,"log_start":"95'515019","ondisk_log_start":"95'515019","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7112040448,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":523022,"num_read_kb":2060488,"num_write":521423,"num_write_kb":2085692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4099,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.678344+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:22:01.678344+0000","last_peered":"2023-10-21T05:22:01.678344+0000","last_clean":"2023-10-21T05:22:01.678344+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:22:01.678344+0000","last_undegraded":"2023-10-21T05:22:01.678344+0000","last_fullsized":"2023-10-21T05:22:01.678344+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'491590","reported_seq":980768,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:04.573826+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:22:04.573826+0000","last_peered":"2023-10-21T05:22:04.573826+0000","last_clean":"2023-10-21T05:22:04.573826+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:22:04.573826+0000","last_undegraded":"2023-10-21T05:22:04.573826+0000","last_fullsized":"2023-10-21T05:22:04.573826+0000","mapping_epoch":16,"log_start":"95'481520","ondisk_log_start":"95'481520","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":417,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 81s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6585769984,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":490052,"num_read_kb":1930456,"num_write":488325,"num_write_kb":1953300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4035,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.678020+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:22:01.678020+0000","last_peered":"2023-10-21T05:22:01.678020+0000","last_clean":"2023-10-21T05:22:01.678020+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:22:01.678020+0000","last_undegraded":"2023-10-21T05:22:01.678020+0000","last_fullsized":"2023-10-21T05:22:01.678020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'511603","reported_seq":1019220,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:01.377412+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:22:01.377412+0000","last_peered":"2023-10-21T05:22:01.377412+0000","last_clean":"2023-10-21T05:22:01.377412+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:22:01.377412+0000","last_undegraded":"2023-10-21T05:22:01.377412+0000","last_fullsized":"2023-10-21T05:22:01.377412+0000","mapping_epoch":16,"log_start":"95'501525","ondisk_log_start":"95'501525","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1240,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 327s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6917062675,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":508743,"num_read_kb":2003784,"num_write":509140,"num_write_kb":2036554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'17759","reported_seq":24430,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.383737+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:22:01.383737+0000","last_peered":"2023-10-21T05:22:01.383737+0000","last_clean":"2023-10-21T05:22:01.383737+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:22:01.383737+0000","last_undegraded":"2023-10-21T05:22:01.383737+0000","last_fullsized":"2023-10-21T05:22:01.383737+0000","mapping_epoch":13,"log_start":"80'7700","ondisk_log_start":"80'7700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":273,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.033081+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:21:00.033081+0000","last_peered":"2023-10-21T05:21:00.033081+0000","last_clean":"2023-10-21T05:21:00.033081+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:21:00.033081+0000","last_undegraded":"2023-10-21T05:21:00.033081+0000","last_fullsized":"2023-10-21T05:21:00.033081+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27608293395,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2038292,"num_read_kb":8029668,"num_write":2036567,"num_write_kb":8146262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16687828992,"data_stored":19581800486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40240,"ondisk_log_size":40240,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38509,"ondisk_log_size":38509,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739338,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9923680,"kb_used_data":8150552,"kb_used_omap":940,"kb_used_meta":1772179,"kb_avail":84448160,"statfs":{"total":96636764160,"available":86474915840,"internally_reserved":0,"allocated":8346165248,"data_stored":9792608201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":1814712161},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.73199999999999998,"15min":0.57399999999999995},"min":{"1min":0.17499999999999999,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":1.849,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.22},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.63200000000000001,"15min":0.53800000000000003},"min":{"1min":0.22700000000000001,"5min":0.192,"15min":0.185},"max":{"1min":1.8080000000000001,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.36399999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:21:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.61899999999999999,"15min":0.59399999999999997},"min":{"1min":0.20599999999999999,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":0.68899999999999995,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.245},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.71299999999999997,"15min":0.57899999999999996},"min":{"1min":0.245,"5min":0.221,"15min":0.19800000000000001},"max":{"1min":0.68600000000000005,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.38800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739335,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5832052,"kb_used_data":4122796,"kb_used_omap":478,"kb_used_meta":1708769,"kb_avail":88539788,"statfs":{"total":96636764160,"available":90664742912,"internally_reserved":0,"allocated":4221743104,"data_stored":4967616539,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1749779570},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,7,10,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:21:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.57699999999999996,"15min":0.53500000000000003},"min":{"1min":0.183,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.59899999999999998,"15min":0.51700000000000002},"min":{"1min":0.254,"5min":0.249,"15min":0.186},"max":{"1min":6.7110000000000003,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.28000000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:21:06 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.378,"15min":0.44500000000000001},"min":{"1min":0.21299999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":0.57699999999999996,"5min":1.73,"15min":8.7669999999999995},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.375,"15min":0.40699999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":1.468,"5min":1.468,"15min":4.306},"last":0.32200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739339,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5387360,"kb_used_data":4024600,"kb_used_omap":504,"kb_used_meta":1362247,"kb_avail":88984480,"statfs":{"total":96636764160,"available":91120107520,"internally_reserved":0,"allocated":4121190400,"data_stored":4822051820,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1394941305},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,3,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:21:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55400000000000005,"15min":0.45000000000000001},"min":{"1min":0.25800000000000001,"5min":0.188,"15min":0.185},"max":{"1min":2.5390000000000001,"5min":15.727,"15min":15.727},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60399999999999998,"15min":0.505},"min":{"1min":0.29099999999999998,"5min":0.247,"15min":0.24399999999999999},"max":{"1min":2.7850000000000001,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:21:20 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.46300000000000002,"15min":0.48899999999999999},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.22},"max":{"1min":0.80300000000000005,"5min":3.831,"15min":6.7430000000000003},"last":0.309},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.61699999999999999,"15min":0.56899999999999995},"min":{"1min":0.27800000000000002,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":2.8450000000000002,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.38100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4120604672,"data_stored":4821540864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4221190144,"data_stored":4967239699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8346034176,"data_stored":9793019923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:22:06.438 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:22:06.438 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:22:15.233 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:22:15.226+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:22:15.234 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:22:15.230+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:22:26.439 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:22:27.006 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:22:27.007 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:22:27.037 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2523,"stamp":"2023-10-21T05:22:26.251290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27634814081,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2064645,"num_read_kb":8074817,"num_write":2044760,"num_write_kb":8178283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78972,"ondisk_log_size":78972,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21387664,"kb_used_data":16356980,"kb_used_omap":1923,"kb_used_meta":5028732,"kb_avail":261727856,"statfs":{"total":289910292480,"available":268009324544,"internally_reserved":0,"allocated":16749547520,"data_stored":19642725328,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1969844,"internal_metadata":5149421900},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,5,1,4,19,15,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16101376,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4515,"num_read_kb":17900,"num_write":4644,"num_write_kb":18576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001980"},"pg_stats":[{"pgid":"2.7","version":"96'14292","reported_seq":19572,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:20.795315+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:22:20.795315+0000","last_peered":"2023-10-21T05:22:20.795315+0000","last_clean":"2023-10-21T05:22:20.795315+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:22:20.795315+0000","last_undegraded":"2023-10-21T05:22:20.795315+0000","last_fullsized":"2023-10-21T05:22:20.795315+0000","mapping_epoch":13,"log_start":"80'4200","ondisk_log_start":"80'4200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7289","reported_seq":10973,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:20.768842+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:22:20.768842+0000","last_peered":"2023-10-21T05:22:20.768842+0000","last_clean":"2023-10-21T05:22:20.768842+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:22:20.768842+0000","last_undegraded":"2023-10-21T05:22:20.768842+0000","last_fullsized":"2023-10-21T05:22:20.768842+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7289,"log_dups_size":0,"ondisk_log_size":7289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3653","reported_seq":5139,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:20.407152+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:22:20.407152+0000","last_peered":"2023-10-21T05:22:20.407152+0000","last_clean":"2023-10-21T05:22:20.407152+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:22:20.407152+0000","last_undegraded":"2023-10-21T05:22:20.407152+0000","last_fullsized":"2023-10-21T05:22:20.407152+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3653,"log_dups_size":0,"ondisk_log_size":3653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7235","reported_seq":10328,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:20.762213+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:22:20.762213+0000","last_peered":"2023-10-21T05:22:20.762213+0000","last_clean":"2023-10-21T05:22:20.762213+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:22:20.762213+0000","last_undegraded":"2023-10-21T05:22:20.762213+0000","last_fullsized":"2023-10-21T05:22:20.762213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7235,"log_dups_size":0,"ondisk_log_size":7235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'523017","reported_seq":1040165,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:24.709313+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:22:24.709313+0000","last_peered":"2023-10-21T05:22:24.709313+0000","last_clean":"2023-10-21T05:22:24.709313+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:22:24.709313+0000","last_undegraded":"2023-10-21T05:22:24.709313+0000","last_fullsized":"2023-10-21T05:22:24.709313+0000","mapping_epoch":16,"log_start":"95'512918","ondisk_log_start":"95'512918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":6999838720,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":518201,"num_read_kb":2041772,"num_write":519526,"num_write_kb":2078104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":227,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.026502+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:21:00.026502+0000","last_peered":"2023-10-21T05:21:00.026502+0000","last_clean":"2023-10-21T05:21:00.026502+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:21:00.026502+0000","last_undegraded":"2023-10-21T05:21:00.026502+0000","last_fullsized":"2023-10-21T05:21:00.026502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'526967","reported_seq":1050818,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:21.474053+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:22:21.474053+0000","last_peered":"2023-10-21T05:22:21.474053+0000","last_clean":"2023-10-21T05:22:21.474053+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:22:21.474053+0000","last_undegraded":"2023-10-21T05:22:21.474053+0000","last_fullsized":"2023-10-21T05:22:21.474053+0000","mapping_epoch":16,"log_start":"95'516919","ondisk_log_start":"95'516919","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7118819328,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":524969,"num_read_kb":2068208,"num_write":523331,"num_write_kb":2093324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4111,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:22.152348+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:22:22.152348+0000","last_peered":"2023-10-21T05:22:22.152348+0000","last_clean":"2023-10-21T05:22:22.152348+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:22:22.152348+0000","last_undegraded":"2023-10-21T05:22:22.152348+0000","last_fullsized":"2023-10-21T05:22:22.152348+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'493417","reported_seq":984372,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:24.709312+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:22:24.709312+0000","last_peered":"2023-10-21T05:22:24.709312+0000","last_clean":"2023-10-21T05:22:24.709312+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:22:24.709312+0000","last_undegraded":"2023-10-21T05:22:24.709312+0000","last_fullsized":"2023-10-21T05:22:24.709312+0000","mapping_epoch":16,"log_start":"95'483320","ondisk_log_start":"95'483320","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":462,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 101s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6591913984,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":491818,"num_read_kb":1937492,"num_write":490152,"num_write_kb":1960608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4047,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:22.152269+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:22:22.152269+0000","last_peered":"2023-10-21T05:22:22.152269+0000","last_clean":"2023-10-21T05:22:22.152269+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:22:22.152269+0000","last_undegraded":"2023-10-21T05:22:22.152269+0000","last_fullsized":"2023-10-21T05:22:22.152269+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'513534","reported_seq":1022985,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:21.464157+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:22:21.464157+0000","last_peered":"2023-10-21T05:22:21.464157+0000","last_clean":"2023-10-21T05:22:21.464157+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:22:21.464157+0000","last_undegraded":"2023-10-21T05:22:21.464157+0000","last_fullsized":"2023-10-21T05:22:21.464157+0000","mapping_epoch":16,"log_start":"95'503526","ondisk_log_start":"95'503526","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1329,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 347s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6923649043,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":510569,"num_read_kb":2011024,"num_write":511013,"num_write_kb":2044046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'17834","reported_seq":24525,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:20.852352+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:22:20.852352+0000","last_peered":"2023-10-21T05:22:20.852352+0000","last_clean":"2023-10-21T05:22:20.852352+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:22:20.852352+0000","last_undegraded":"2023-10-21T05:22:20.852352+0000","last_fullsized":"2023-10-21T05:22:20.852352+0000","mapping_epoch":13,"log_start":"80'7800","ondisk_log_start":"80'7800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":273,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.033081+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:21:00.033081+0000","last_peered":"2023-10-21T05:21:00.033081+0000","last_clean":"2023-10-21T05:21:00.033081+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:21:00.033081+0000","last_undegraded":"2023-10-21T05:21:00.033081+0000","last_fullsized":"2023-10-21T05:21:00.033081+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27634221075,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2045557,"num_read_kb":8058496,"num_write":2044022,"num_write_kb":8176082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16748662784,"data_stored":19642634278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40252,"ondisk_log_size":40252,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38621,"ondisk_log_size":38621,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739342,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10021156,"kb_used_data":8179996,"kb_used_omap":940,"kb_used_meta":1840211,"kb_avail":84350684,"statfs":{"total":96636764160,"available":86375100416,"internally_reserved":0,"allocated":8376315904,"data_stored":9822758857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":1884376929},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.64600000000000002,"15min":0.57799999999999996},"min":{"1min":0.22,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.68600000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.44},{"interface":"front","average":{"1min":0.309,"5min":0.52200000000000002,"15min":0.53800000000000003},"min":{"1min":0.19700000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.501,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.29999999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.60199999999999998,"15min":0.59099999999999997},"min":{"1min":0.218,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":3.9990000000000001,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.63,"15min":0.56999999999999995},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.19800000000000001},"max":{"1min":0.59199999999999997,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.41399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739339,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5842180,"kb_used_data":4137784,"kb_used_omap":478,"kb_used_meta":1703905,"kb_avail":88529660,"statfs":{"total":96636764160,"available":90654371840,"internally_reserved":0,"allocated":4237090816,"data_stored":4982964251,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1744798834},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,0,1,6,11,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.57799999999999996,"15min":0.52800000000000002},"min":{"1min":0.192,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":1.756,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.59499999999999997,"15min":0.51000000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.186},"max":{"1min":1.536,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.374}]},{"osd":2,"last update":"Sat Oct 21 05:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.379,"15min":0.42799999999999999},"min":{"1min":0.222,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":0.63300000000000001,"5min":1.73,"15min":8.7669999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.376,"15min":0.39200000000000002},"min":{"1min":0.25600000000000001,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":1.002,"5min":1.468,"15min":4.1920000000000002},"last":0.41899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739343,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5524328,"kb_used_data":4039200,"kb_used_omap":504,"kb_used_meta":1484615,"kb_avail":88847512,"statfs":{"total":96636764160,"available":90979852288,"internally_reserved":0,"allocated":4136140800,"data_stored":4837002220,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1520246137},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,11,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:22:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.54400000000000004,"15min":0.44800000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.86599999999999999,"5min":15.727,"15min":15.727},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.59299999999999997,"15min":0.504},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.83399999999999996,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.54700000000000004}]},{"osd":2,"last update":"Sat Oct 21 05:22:20 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.56000000000000005,"15min":0.52800000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.71799999999999997,"15min":0.60899999999999999},"min":{"1min":0.28299999999999997,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":9.8680000000000003,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4135645184,"data_stored":4836581376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4236668928,"data_stored":4982718483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8376348672,"data_stored":9823334419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:22:27.039 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:22:27.040 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:22:47.041 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:22:47.549 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:22:47.551 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:22:47.591 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2533,"stamp":"2023-10-21T05:22:46.255802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27660213377,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2072048,"num_read_kb":8104277,"num_write":2052096,"num_write_kb":8207627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79003,"ondisk_log_size":79003,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21527488,"kb_used_data":16416488,"kb_used_omap":1923,"kb_used_meta":5109052,"kb_avail":261588032,"statfs":{"total":289910292480,"available":267866144768,"internally_reserved":0,"allocated":16810483712,"data_stored":19703661520,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1969844,"internal_metadata":5231669580},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,5,11,13,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16035840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4651,"num_read_kb":18488,"num_write":4645,"num_write_kb":18580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001965"},"pg_stats":[{"pgid":"2.7","version":"96'14364","reported_seq":19660,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:41.313618+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:22:41.313618+0000","last_peered":"2023-10-21T05:22:41.313618+0000","last_clean":"2023-10-21T05:22:41.313618+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:22:41.313618+0000","last_undegraded":"2023-10-21T05:22:41.313618+0000","last_fullsized":"2023-10-21T05:22:41.313618+0000","mapping_epoch":13,"log_start":"80'4300","ondisk_log_start":"80'4300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7325","reported_seq":11017,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:41.302366+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:22:41.302366+0000","last_peered":"2023-10-21T05:22:41.302366+0000","last_clean":"2023-10-21T05:22:41.302366+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:22:41.302366+0000","last_undegraded":"2023-10-21T05:22:41.302366+0000","last_fullsized":"2023-10-21T05:22:41.302366+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7325,"log_dups_size":0,"ondisk_log_size":7325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3671","reported_seq":5161,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:41.188156+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:22:41.188156+0000","last_peered":"2023-10-21T05:22:41.188156+0000","last_clean":"2023-10-21T05:22:41.188156+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:22:41.188156+0000","last_undegraded":"2023-10-21T05:22:41.188156+0000","last_fullsized":"2023-10-21T05:22:41.188156+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3671,"log_dups_size":0,"ondisk_log_size":3671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7271","reported_seq":10372,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:41.302406+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:22:41.302406+0000","last_peered":"2023-10-21T05:22:41.302406+0000","last_clean":"2023-10-21T05:22:41.302406+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:22:41.302406+0000","last_undegraded":"2023-10-21T05:22:41.302406+0000","last_fullsized":"2023-10-21T05:22:41.302406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7271,"log_dups_size":0,"ondisk_log_size":7271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'524888","reported_seq":1043943,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:44.807051+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:22:44.807051+0000","last_peered":"2023-10-21T05:22:44.807051+0000","last_clean":"2023-10-21T05:22:44.807051+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:22:44.807051+0000","last_undegraded":"2023-10-21T05:22:44.807051+0000","last_fullsized":"2023-10-21T05:22:44.807051+0000","mapping_epoch":16,"log_start":"95'514818","ondisk_log_start":"95'514818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7006269440,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":520104,"num_read_kb":2049328,"num_write":521397,"num_write_kb":2085588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":227,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.026502+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:21:00.026502+0000","last_peered":"2023-10-21T05:21:00.026502+0000","last_clean":"2023-10-21T05:21:00.026502+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:21:00.026502+0000","last_undegraded":"2023-10-21T05:21:00.026502+0000","last_fullsized":"2023-10-21T05:21:00.026502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'528892","reported_seq":1054660,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:41.571913+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:22:41.571913+0000","last_peered":"2023-10-21T05:22:41.571913+0000","last_clean":"2023-10-21T05:22:41.571913+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:22:41.571913+0000","last_undegraded":"2023-10-21T05:22:41.571913+0000","last_fullsized":"2023-10-21T05:22:41.571913+0000","mapping_epoch":16,"log_start":"95'518819","ondisk_log_start":"95'518819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7125594112,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":526888,"num_read_kb":2075824,"num_write":525256,"num_write_kb":2101024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4123,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:42.657462+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:22:42.657462+0000","last_peered":"2023-10-21T05:22:42.657462+0000","last_clean":"2023-10-21T05:22:42.657462+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:22:42.657462+0000","last_undegraded":"2023-10-21T05:22:42.657462+0000","last_fullsized":"2023-10-21T05:22:42.657462+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'495199","reported_seq":987916,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:44.792069+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:22:44.792069+0000","last_peered":"2023-10-21T05:22:44.792069+0000","last_clean":"2023-10-21T05:22:44.792069+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:22:44.792069+0000","last_undegraded":"2023-10-21T05:22:44.792069+0000","last_fullsized":"2023-10-21T05:22:44.792069+0000","mapping_epoch":16,"log_start":"95'485120","ondisk_log_start":"95'485120","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":518,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 121s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6597967872,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":493572,"num_read_kb":1944488,"num_write":491931,"num_write_kb":1967724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4059,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:42.651177+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:22:42.651177+0000","last_peered":"2023-10-21T05:22:42.651177+0000","last_clean":"2023-10-21T05:22:42.651177+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:22:42.651177+0000","last_undegraded":"2023-10-21T05:22:42.651177+0000","last_fullsized":"2023-10-21T05:22:42.651177+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'515335","reported_seq":1026626,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:22:41.568686+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:22:41.568686+0000","last_peered":"2023-10-21T05:22:41.568686+0000","last_clean":"2023-10-21T05:22:41.568686+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:22:41.568686+0000","last_undegraded":"2023-10-21T05:22:41.568686+0000","last_fullsized":"2023-10-21T05:22:41.568686+0000","mapping_epoch":16,"log_start":"95'505326","ondisk_log_start":"95'505326","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1392,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 368s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6929788947,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":512396,"num_read_kb":2018316,"num_write":512774,"num_write_kb":2051090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'17924","reported_seq":24635,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:41.335546+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:22:41.335546+0000","last_peered":"2023-10-21T05:22:41.335546+0000","last_clean":"2023-10-21T05:22:41.335546+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:22:41.335546+0000","last_undegraded":"2023-10-21T05:22:41.335546+0000","last_fullsized":"2023-10-21T05:22:41.335546+0000","mapping_epoch":13,"log_start":"80'7900","ondisk_log_start":"80'7900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":273,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.033081+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:21:00.033081+0000","last_peered":"2023-10-21T05:21:00.033081+0000","last_clean":"2023-10-21T05:21:00.033081+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:21:00.033081+0000","last_undegraded":"2023-10-21T05:21:00.033081+0000","last_fullsized":"2023-10-21T05:21:00.033081+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27659620371,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2052960,"num_read_kb":8087956,"num_write":2051358,"num_write_kb":8205426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16809205760,"data_stored":19703177254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40231,"ondisk_log_size":40231,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38673,"ondisk_log_size":38673,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739346,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10125984,"kb_used_data":8209560,"kb_used_omap":940,"kb_used_meta":1915475,"kb_avail":84245856,"statfs":{"total":96636764160,"available":86267756544,"internally_reserved":0,"allocated":8406589440,"data_stored":9853032393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":1961447265},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.64600000000000002,"15min":0.57799999999999996},"min":{"1min":0.22,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.68600000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.309,"5min":0.52200000000000002,"15min":0.53800000000000003},"min":{"1min":0.19700000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.501,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.34200000000000003}]},{"osd":1,"last update":"Sat Oct 21 05:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.60199999999999998,"15min":0.59099999999999997},"min":{"1min":0.218,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":3.9990000000000001,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.63,"15min":0.56999999999999995},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.19800000000000001},"max":{"1min":0.59199999999999997,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.36799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739343,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5929740,"kb_used_data":4153156,"kb_used_omap":478,"kb_used_meta":1776097,"kb_avail":88442100,"statfs":{"total":96636764160,"available":90564710400,"internally_reserved":0,"allocated":4252831744,"data_stored":4998705179,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1818723442},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,6,4,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.57799999999999996,"15min":0.52800000000000002},"min":{"1min":0.192,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":1.756,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.59499999999999997,"15min":0.51000000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.186},"max":{"1min":1.536,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.51200000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.379,"15min":0.42799999999999999},"min":{"1min":0.222,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":0.63300000000000001,"5min":1.73,"15min":8.7669999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.372,"5min":0.376,"15min":0.39200000000000002},"min":{"1min":0.25600000000000001,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":1.002,"5min":1.468,"15min":4.1920000000000002},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359739347,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5471764,"kb_used_data":4053772,"kb_used_omap":504,"kb_used_meta":1417479,"kb_avail":88900076,"statfs":{"total":96636764160,"available":91033677824,"internally_reserved":0,"allocated":4151062528,"data_stored":4851923948,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1451498873},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,3,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:22:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.54400000000000004,"15min":0.44800000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.86599999999999999,"5min":15.727,"15min":15.727},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.59299999999999997,"15min":0.504},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.83399999999999996,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:22:20 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.56000000000000005,"15min":0.52800000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.71799999999999997,"15min":0.60899999999999999},"min":{"1min":0.28299999999999997,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":9.8680000000000003,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.41699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4150603776,"data_stored":4851539968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4251766784,"data_stored":4997816339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8406835200,"data_stored":9853820947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:22:47.593 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:22:47.593 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:22:55.645 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:22:55.638+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:22:55.657 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:22:55.650+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:22:55.657 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:22:55.650+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:22:55.675 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:22:55.670+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:22:55.675 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:22:55.670+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:23:07.594 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:23:08.186 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:23:08.188 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:23:08.221 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2543,"stamp":"2023-10-21T05:23:06.259340+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27685858433,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2079440,"num_read_kb":8133693,"num_write":2059510,"num_write_kb":8237283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79095,"ondisk_log_size":79095,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21815872,"kb_used_data":16476324,"kb_used_omap":1923,"kb_used_meta":5337596,"kb_avail":261299648,"statfs":{"total":289910292480,"available":267570839552,"internally_reserved":0,"allocated":16871755776,"data_stored":19764933584,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1969844,"internal_metadata":5465698636},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,4,17,18,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15982592,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4497,"num_read_kb":17884,"num_write":4616,"num_write_kb":18464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002099"},"pg_stats":[{"pgid":"2.7","version":"96'14432","reported_seq":19744,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:59.298950+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:22:59.298950+0000","last_peered":"2023-10-21T05:22:59.298950+0000","last_clean":"2023-10-21T05:22:59.298950+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:22:59.298950+0000","last_undegraded":"2023-10-21T05:22:59.298950+0000","last_fullsized":"2023-10-21T05:22:59.298950+0000","mapping_epoch":13,"log_start":"80'4400","ondisk_log_start":"80'4400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7359","reported_seq":11059,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:59.279968+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:22:59.279968+0000","last_peered":"2023-10-21T05:22:59.279968+0000","last_clean":"2023-10-21T05:22:59.279968+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:22:59.279968+0000","last_undegraded":"2023-10-21T05:22:59.279968+0000","last_fullsized":"2023-10-21T05:22:59.279968+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7359,"log_dups_size":0,"ondisk_log_size":7359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3688","reported_seq":5182,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:58.513210+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:22:58.513210+0000","last_peered":"2023-10-21T05:22:58.513210+0000","last_clean":"2023-10-21T05:22:58.513210+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:22:58.513210+0000","last_undegraded":"2023-10-21T05:22:58.513210+0000","last_fullsized":"2023-10-21T05:22:58.513210+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7305","reported_seq":10414,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:59.280436+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:22:59.280436+0000","last_peered":"2023-10-21T05:22:59.280436+0000","last_clean":"2023-10-21T05:22:59.280436+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:22:59.280436+0000","last_undegraded":"2023-10-21T05:22:59.280436+0000","last_fullsized":"2023-10-21T05:22:59.280436+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7305,"log_dups_size":0,"ondisk_log_size":7305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'526789","reported_seq":1047737,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:04.935768+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:23:04.935768+0000","last_peered":"2023-10-21T05:23:04.935768+0000","last_clean":"2023-10-21T05:23:04.935768+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:23:04.935768+0000","last_undegraded":"2023-10-21T05:23:04.935768+0000","last_fullsized":"2023-10-21T05:23:04.935768+0000","mapping_epoch":16,"log_start":"96'516718","ondisk_log_start":"96'516718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7012806656,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":521995,"num_read_kb":2056864,"num_write":523298,"num_write_kb":2093192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":227,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.026502+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:21:00.026502+0000","last_peered":"2023-10-21T05:21:00.026502+0000","last_clean":"2023-10-21T05:21:00.026502+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:21:00.026502+0000","last_undegraded":"2023-10-21T05:21:00.026502+0000","last_fullsized":"2023-10-21T05:21:00.026502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'530801","reported_seq":1058492,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:01.714619+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:23:01.714619+0000","last_peered":"2023-10-21T05:23:01.714619+0000","last_clean":"2023-10-21T05:23:01.714619+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:23:01.714619+0000","last_undegraded":"2023-10-21T05:23:01.714619+0000","last_fullsized":"2023-10-21T05:23:01.714619+0000","mapping_epoch":16,"log_start":"96'520719","ondisk_log_start":"96'520719","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7132372992,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":528810,"num_read_kb":2083488,"num_write":527165,"num_write_kb":2108660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4136,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:03.138940+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:23:03.138940+0000","last_peered":"2023-10-21T05:23:03.138940+0000","last_clean":"2023-10-21T05:23:03.138940+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:23:03.138940+0000","last_undegraded":"2023-10-21T05:23:03.138940+0000","last_fullsized":"2023-10-21T05:23:03.138940+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'496988","reported_seq":991495,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:04.954546+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:23:04.954546+0000","last_peered":"2023-10-21T05:23:04.954546+0000","last_clean":"2023-10-21T05:23:04.954546+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:23:04.954546+0000","last_undegraded":"2023-10-21T05:23:04.954546+0000","last_fullsized":"2023-10-21T05:23:04.954546+0000","mapping_epoch":16,"log_start":"96'486920","ondisk_log_start":"96'486920","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":566,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 142s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6604009472,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":495357,"num_read_kb":1951572,"num_write":493720,"num_write_kb":1974880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4071,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:03.116206+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:23:03.116206+0000","last_peered":"2023-10-21T05:23:03.116206+0000","last_clean":"2023-10-21T05:23:03.116206+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:23:03.116206+0000","last_undegraded":"2023-10-21T05:23:03.116206+0000","last_fullsized":"2023-10-21T05:23:03.116206+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'517191","reported_seq":1030278,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:01.655319+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:23:01.655319+0000","last_peered":"2023-10-21T05:23:01.655319+0000","last_clean":"2023-10-21T05:23:01.655319+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:23:01.655319+0000","last_undegraded":"2023-10-21T05:23:01.655319+0000","last_fullsized":"2023-10-21T05:23:01.655319+0000","mapping_epoch":16,"log_start":"96'507126","ondisk_log_start":"96'507126","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1454,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 388s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6936076307,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":514190,"num_read_kb":2025448,"num_write":514589,"num_write_kb":2058350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18008","reported_seq":24739,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:59.323034+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:22:59.323034+0000","last_peered":"2023-10-21T05:22:59.323034+0000","last_clean":"2023-10-21T05:22:59.323034+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:22:59.323034+0000","last_undegraded":"2023-10-21T05:22:59.323034+0000","last_fullsized":"2023-10-21T05:22:59.323034+0000","mapping_epoch":13,"log_start":"80'8000","ondisk_log_start":"80'8000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":273,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.033081+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:21:00.033081+0000","last_peered":"2023-10-21T05:21:00.033081+0000","last_clean":"2023-10-21T05:21:00.033081+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:21:00.033081+0000","last_undegraded":"2023-10-21T05:21:00.033081+0000","last_fullsized":"2023-10-21T05:21:00.033081+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27685265427,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2060352,"num_read_kb":8117372,"num_write":2058772,"num_write_kb":8235082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16869720064,"data_stored":19763691558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40286,"ondisk_log_size":40286,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38710,"ondisk_log_size":38710,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739350,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10277300,"kb_used_data":8240104,"kb_used_omap":940,"kb_used_meta":2036243,"kb_avail":84094540,"statfs":{"total":96636764160,"available":86112808960,"internally_reserved":0,"allocated":8437866496,"data_stored":9884309449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":2085113697},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.64600000000000002,"15min":0.57799999999999996},"min":{"1min":0.22,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.68600000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.309,"5min":0.52200000000000002,"15min":0.53800000000000003},"min":{"1min":0.19700000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.501,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.25600000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:22:10 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.60199999999999998,"15min":0.59099999999999997},"min":{"1min":0.218,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":3.9990000000000001,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.63,"15min":0.56999999999999995},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.19800000000000001},"max":{"1min":0.59199999999999997,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.437}]}]},{"osd":1,"up_from":8,"seq":34359739347,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5998032,"kb_used_data":4167816,"kb_used_omap":478,"kb_used_meta":1829729,"kb_avail":88373808,"statfs":{"total":96636764160,"available":90494779392,"internally_reserved":0,"allocated":4267843584,"data_stored":5013717019,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1873642610},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,14,14,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.57799999999999996,"15min":0.52800000000000002},"min":{"1min":0.192,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":1.756,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.59499999999999997,"15min":0.51000000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.186},"max":{"1min":1.536,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Sat Oct 21 05:22:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.379,"15min":0.42799999999999999},"min":{"1min":0.222,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":0.63300000000000001,"5min":1.73,"15min":8.7669999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.376,"15min":0.39200000000000002},"min":{"1min":0.25600000000000001,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":1.002,"5min":1.468,"15min":4.1920000000000002},"last":0.312}]}]},{"osd":0,"up_from":8,"seq":34359739351,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5540540,"kb_used_data":4068404,"kb_used_omap":504,"kb_used_meta":1471623,"kb_avail":88831300,"statfs":{"total":96636764160,"available":90963251200,"internally_reserved":0,"allocated":4166045696,"data_stored":4866907116,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1506942329},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,0,2,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:22:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.54400000000000004,"15min":0.44800000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.86599999999999999,"5min":15.727,"15min":15.727},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.59299999999999997,"15min":0.504},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.83399999999999996,"5min":15.757999999999999,"15min":15.757999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:22:20 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.56000000000000005,"15min":0.52800000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":9.3109999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.71799999999999997,"15min":0.60899999999999999},"min":{"1min":0.28299999999999997,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":9.8680000000000003,"5min":10.967000000000001,"15min":10.967000000000001},"last":0.39300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4165705728,"data_stored":4866641920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4267020288,"data_stored":5013069843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8436994048,"data_stored":9883979795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:23:08.223 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:23:08.224 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:23:08.757 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:23:08.794 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:23:12.676 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:12.666+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:23:12.676 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:12.666+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:23:12.677 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:12.666+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:23:12.677 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:12.666+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:23:12.677 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:12.666+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:23:12.686 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:12.670+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:23:12.687 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:12.678+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:23:12.687 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:12.678+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:23:12.687 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:12.678+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:23:26.911 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T05:23:26.912 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:23:26.912 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:23:26.913 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:23:26.913 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:23:26.913 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T05:23:26.914 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:23:26.914 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:23:26.914 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:23:26.914 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:23:26.915 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:23:26.915 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:23:26.915 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:23:26.916 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:26.906+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:23:28.795 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:23:29.416 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:23:29.416 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:23:29.443 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2554,"stamp":"2023-10-21T05:23:28.263056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27714989185,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2087989,"num_read_kb":8167713,"num_write":2067967,"num_write_kb":8271111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79137,"ondisk_log_size":79137,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22055928,"kb_used_data":16540060,"kb_used_omap":1923,"kb_used_meta":5513916,"kb_avail":261059592,"statfs":{"total":289910292480,"available":267325022208,"internally_reserved":0,"allocated":16937021440,"data_stored":19830199248,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1969844,"internal_metadata":5646250316},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,7,8,22,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16338944,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4826,"num_read_kb":19208,"num_write":4743,"num_write_kb":18972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002054"},"pg_stats":[{"pgid":"2.7","version":"96'14472","reported_seq":19804,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:26.887383+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:23:26.887383+0000","last_peered":"2023-10-21T05:23:26.887383+0000","last_clean":"2023-10-21T05:23:26.887383+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:23:26.887383+0000","last_undegraded":"2023-10-21T05:23:26.887383+0000","last_fullsized":"2023-10-21T05:23:26.887383+0000","mapping_epoch":13,"log_start":"80'4400","ondisk_log_start":"80'4400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7379","reported_seq":11089,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:26.865554+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:23:26.865554+0000","last_peered":"2023-10-21T05:23:26.865554+0000","last_clean":"2023-10-21T05:23:26.865554+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:23:26.865554+0000","last_undegraded":"2023-10-21T05:23:26.865554+0000","last_fullsized":"2023-10-21T05:23:26.865554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7379,"log_dups_size":0,"ondisk_log_size":7379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3698","reported_seq":5198,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:26.348634+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:23:26.348634+0000","last_peered":"2023-10-21T05:23:26.348634+0000","last_clean":"2023-10-21T05:23:26.348634+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:23:26.348634+0000","last_undegraded":"2023-10-21T05:23:26.348634+0000","last_fullsized":"2023-10-21T05:23:26.348634+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3698,"log_dups_size":0,"ondisk_log_size":3698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7325","reported_seq":10445,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:26.864264+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:23:26.864264+0000","last_peered":"2023-10-21T05:23:26.864264+0000","last_clean":"2023-10-21T05:23:26.864264+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:23:26.864264+0000","last_undegraded":"2023-10-21T05:23:26.864264+0000","last_fullsized":"2023-10-21T05:23:26.864264+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7325,"log_dups_size":0,"ondisk_log_size":7325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'528690","reported_seq":1051518,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:25.003291+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:23:25.003291+0000","last_peered":"2023-10-21T05:23:25.003291+0000","last_clean":"2023-10-21T05:23:25.003291+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:23:25.003291+0000","last_undegraded":"2023-10-21T05:23:25.003291+0000","last_fullsized":"2023-10-21T05:23:25.003291+0000","mapping_epoch":16,"log_start":"96'518618","ondisk_log_start":"96'518618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7019270144,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":523881,"num_read_kb":2064372,"num_write":525199,"num_write_kb":2100796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":228,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:14.343638+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:21:14.343638+0000","last_peered":"2023-10-21T05:21:14.343638+0000","last_clean":"2023-10-21T05:21:14.343638+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:21:14.343638+0000","last_undegraded":"2023-10-21T05:21:14.343638+0000","last_fullsized":"2023-10-21T05:21:14.343638+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'533224","reported_seq":1063363,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:26.857119+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:23:26.857119+0000","last_peered":"2023-10-21T05:23:26.857119+0000","last_clean":"2023-10-21T05:23:26.857119+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:23:26.857119+0000","last_undegraded":"2023-10-21T05:23:26.857119+0000","last_fullsized":"2023-10-21T05:23:26.857119+0000","mapping_epoch":16,"log_start":"96'523219","ondisk_log_start":"96'523219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7140872192,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":531257,"num_read_kb":2093208,"num_write":529588,"num_write_kb":2118352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4148,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:23.623369+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:23:23.623369+0000","last_peered":"2023-10-21T05:23:23.623369+0000","last_clean":"2023-10-21T05:23:23.623369+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:23:23.623369+0000","last_undegraded":"2023-10-21T05:23:23.623369+0000","last_fullsized":"2023-10-21T05:23:23.623369+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'498801","reported_seq":995139,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:25.017641+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:23:25.017641+0000","last_peered":"2023-10-21T05:23:25.017641+0000","last_clean":"2023-10-21T05:23:25.017641+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:23:25.017641+0000","last_undegraded":"2023-10-21T05:23:25.017641+0000","last_fullsized":"2023-10-21T05:23:25.017641+0000","mapping_epoch":16,"log_start":"96'488720","ondisk_log_start":"96'488720","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":643,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 162s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6610104320,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":497172,"num_read_kb":1958804,"num_write":495527,"num_write_kb":1982108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4083,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:23.592264+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:23:23.592264+0000","last_peered":"2023-10-21T05:23:23.592264+0000","last_clean":"2023-10-21T05:23:23.592264+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:23:23.592264+0000","last_undegraded":"2023-10-21T05:23:23.592264+0000","last_fullsized":"2023-10-21T05:23:23.592264+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'519556","reported_seq":1035061,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:26.893710+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:23:26.893710+0000","last_peered":"2023-10-21T05:23:26.893710+0000","last_clean":"2023-10-21T05:23:26.893710+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:23:26.893710+0000","last_undegraded":"2023-10-21T05:23:26.893710+0000","last_fullsized":"2023-10-21T05:23:26.893710+0000","mapping_epoch":16,"log_start":"96'509526","ondisk_log_start":"96'509526","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1517,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 413s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6944149523,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":516591,"num_read_kb":2035008,"num_write":516915,"num_write_kb":2067654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18058","reported_seq":24813,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:26.897968+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:23:26.897968+0000","last_peered":"2023-10-21T05:23:26.897968+0000","last_clean":"2023-10-21T05:23:26.897968+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:23:26.897968+0000","last_undegraded":"2023-10-21T05:23:26.897968+0000","last_fullsized":"2023-10-21T05:23:26.897968+0000","mapping_epoch":13,"log_start":"80'8000","ondisk_log_start":"80'8000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":273,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.033081+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:21:00.033081+0000","last_peered":"2023-10-21T05:21:00.033081+0000","last_clean":"2023-10-21T05:21:00.033081+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:21:00.033081+0000","last_undegraded":"2023-10-21T05:21:00.033081+0000","last_fullsized":"2023-10-21T05:21:00.033081+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27714396179,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2068901,"num_read_kb":8151392,"num_write":2067229,"num_write_kb":8268910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16935129088,"data_stored":19829100582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40188,"ondisk_log_size":40188,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38850,"ondisk_log_size":38850,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739354,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10393404,"kb_used_data":8270128,"kb_used_omap":940,"kb_used_meta":2122323,"kb_avail":83978436,"statfs":{"total":96636764160,"available":85993918464,"internally_reserved":0,"allocated":8468611072,"data_stored":9915054025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":2173259617},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:11 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.64400000000000002,"15min":0.58699999999999997},"min":{"1min":0.19600000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":3.4889999999999999,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.307},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.58499999999999996,"15min":0.56799999999999995},"min":{"1min":0.22700000000000001,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":4.4880000000000004,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.32200000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:23:11 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.625,"15min":0.59699999999999998},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":3.855,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.63200000000000001,"15min":0.57099999999999995},"min":{"1min":0.247,"5min":0.221,"15min":0.19800000000000001},"max":{"1min":0.54200000000000004,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739352,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6057244,"kb_used_data":4186772,"kb_used_omap":478,"kb_used_meta":1869985,"kb_avail":88314596,"statfs":{"total":96636764160,"available":90434146304,"internally_reserved":0,"allocated":4287254528,"data_stored":5033127963,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1914864754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,0,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:12 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51100000000000001,"15min":0.52400000000000002},"min":{"1min":0.23400000000000001,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":0.84899999999999998,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.51600000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.88700000000000001,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":1.591}]},{"osd":2,"last update":"Sat Oct 21 05:23:12 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.36099999999999999,"15min":0.41499999999999998},"min":{"1min":0.224,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":0.59299999999999997,"5min":0.79200000000000004,"15min":8.7669999999999995},"last":0.313},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.378,"15min":0.38800000000000001},"min":{"1min":0.23499999999999999,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":0.55600000000000005,"5min":1.468,"15min":4.1920000000000002},"last":1.3919999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739355,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5605280,"kb_used_data":4083160,"kb_used_omap":504,"kb_used_meta":1521607,"kb_avail":88766560,"statfs":{"total":96636764160,"available":90896957440,"internally_reserved":0,"allocated":4181155840,"data_stored":4882017260,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1558125945},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,1,19,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:23:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":0.56399999999999995,"15min":0.495},"min":{"1min":0.22800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":6.0030000000000001,"5min":6.0030000000000001,"15min":15.727},"last":0.26300000000000001},{"interface":"front","average":{"1min":1.5409999999999999,"5min":0.69799999999999995,"15min":0.57599999999999996},"min":{"1min":0.29399999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":6.0529999999999999,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:23:21 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":0.69799999999999995,"15min":0.56899999999999995},"min":{"1min":0.27600000000000002,"5min":0.22,"15min":0.22},"max":{"1min":5.3799999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":0.78800000000000003,"15min":0.66300000000000003},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":5.3200000000000003,"5min":9.8680000000000003,"15min":10.967000000000001},"last":0.39700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4180910080,"data_stored":4881846272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4286476288,"data_stored":5032525843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8467742720,"data_stored":9914728467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:23:29.444 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:23:29.445 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:23:33.230 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:33.222+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:23:33.230 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:33.222+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:23:33.231 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:33.222+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:23:33.231 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:33.222+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:23:33.244 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:33.238+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:23:33.248 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:33.242+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:23:33.270 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:33.266+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:23:33.271 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:33.266+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:23:34.255 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:34.246+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:23:34.267 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:34.258+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:23:49.446 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:23:50.013 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:23:50.014 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:23:50.062 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2564,"stamp":"2023-10-21T05:23:48.266553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27740122241,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2095210,"num_read_kb":8196429,"num_write":2075245,"num_write_kb":8300223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79113,"ondisk_log_size":79113,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22190400,"kb_used_data":16598056,"kb_used_omap":1923,"kb_used_meta":5590396,"kb_avail":260925120,"statfs":{"total":289910292480,"available":267187322880,"internally_reserved":0,"allocated":16996409344,"data_stored":19889587152,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1969844,"internal_metadata":5724565836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1,7,25,13,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15679488,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4433,"num_read_kb":17628,"num_write":4511,"num_write_kb":18044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002065"},"pg_stats":[{"pgid":"2.7","version":"96'14514","reported_seq":19862,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:45.676650+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:23:45.676650+0000","last_peered":"2023-10-21T05:23:45.676650+0000","last_clean":"2023-10-21T05:23:45.676650+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:23:45.676650+0000","last_undegraded":"2023-10-21T05:23:45.676650+0000","last_fullsized":"2023-10-21T05:23:45.676650+0000","mapping_epoch":13,"log_start":"80'4500","ondisk_log_start":"80'4500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7402","reported_seq":11120,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:45.663259+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:23:45.663259+0000","last_peered":"2023-10-21T05:23:45.663259+0000","last_clean":"2023-10-21T05:23:45.663259+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:23:45.663259+0000","last_undegraded":"2023-10-21T05:23:45.663259+0000","last_fullsized":"2023-10-21T05:23:45.663259+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7402,"log_dups_size":0,"ondisk_log_size":7402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3710","reported_seq":5214,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:45.459497+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:23:45.459497+0000","last_peered":"2023-10-21T05:23:45.459497+0000","last_clean":"2023-10-21T05:23:45.459497+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:23:45.459497+0000","last_undegraded":"2023-10-21T05:23:45.459497+0000","last_fullsized":"2023-10-21T05:23:45.459497+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3710,"log_dups_size":0,"ondisk_log_size":3710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7348","reported_seq":10476,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:45.663290+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:23:45.663290+0000","last_peered":"2023-10-21T05:23:45.663290+0000","last_clean":"2023-10-21T05:23:45.663290+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:23:45.663290+0000","last_undegraded":"2023-10-21T05:23:45.663290+0000","last_fullsized":"2023-10-21T05:23:45.663290+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7348,"log_dups_size":0,"ondisk_log_size":7348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'530607","reported_seq":1055240,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:45.173482+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:23:45.173482+0000","last_peered":"2023-10-21T05:23:45.173482+0000","last_clean":"2023-10-21T05:23:45.173482+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:23:45.173482+0000","last_undegraded":"2023-10-21T05:23:45.173482+0000","last_fullsized":"2023-10-21T05:23:45.173482+0000","mapping_epoch":16,"log_start":"96'520518","ondisk_log_start":"96'520518","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7025823744,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":525683,"num_read_kb":2071536,"num_write":527116,"num_write_kb":2108464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":228,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:14.343638+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:21:14.343638+0000","last_peered":"2023-10-21T05:21:14.343638+0000","last_clean":"2023-10-21T05:21:14.343638+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:21:14.343638+0000","last_undegraded":"2023-10-21T05:21:14.343638+0000","last_fullsized":"2023-10-21T05:21:14.343638+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'535164","reported_seq":1067150,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:46.991207+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:23:46.991207+0000","last_peered":"2023-10-21T05:23:46.991207+0000","last_clean":"2023-10-21T05:23:46.991207+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:23:46.991207+0000","last_undegraded":"2023-10-21T05:23:46.991207+0000","last_fullsized":"2023-10-21T05:23:46.991207+0000","mapping_epoch":16,"log_start":"96'525119","ondisk_log_start":"96'525119","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7147655168,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":533107,"num_read_kb":2100568,"num_write":531528,"num_write_kb":2126112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4160,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:44.085969+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:23:44.085969+0000","last_peered":"2023-10-21T05:23:44.085969+0000","last_clean":"2023-10-21T05:23:44.085969+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:23:44.085969+0000","last_undegraded":"2023-10-21T05:23:44.085969+0000","last_fullsized":"2023-10-21T05:23:44.085969+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'500469","reported_seq":998590,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:45.172678+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:23:45.172678+0000","last_peered":"2023-10-21T05:23:45.172678+0000","last_clean":"2023-10-21T05:23:45.172678+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:23:45.172678+0000","last_undegraded":"2023-10-21T05:23:45.172678+0000","last_fullsized":"2023-10-21T05:23:45.172678+0000","mapping_epoch":16,"log_start":"96'490420","ondisk_log_start":"96'490420","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":778,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 182s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6615711744,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":498941,"num_read_kb":1965836,"num_write":497190,"num_write_kb":1988760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4095,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:44.106503+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:23:44.106503+0000","last_peered":"2023-10-21T05:23:44.106503+0000","last_clean":"2023-10-21T05:23:44.106503+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:23:44.106503+0000","last_undegraded":"2023-10-21T05:23:44.106503+0000","last_fullsized":"2023-10-21T05:23:44.106503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'521354","reported_seq":1038667,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:23:46.991381+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:23:46.991381+0000","last_peered":"2023-10-21T05:23:46.991381+0000","last_clean":"2023-10-21T05:23:46.991381+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:23:46.991381+0000","last_undegraded":"2023-10-21T05:23:46.991381+0000","last_fullsized":"2023-10-21T05:23:46.991381+0000","mapping_epoch":16,"log_start":"96'511326","ondisk_log_start":"96'511326","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1585,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 433s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6950338579,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":518391,"num_read_kb":2042168,"num_write":518673,"num_write_kb":2074686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18111","reported_seq":24888,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:45.719938+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:23:45.719938+0000","last_peered":"2023-10-21T05:23:45.719938+0000","last_clean":"2023-10-21T05:23:45.719938+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:23:45.719938+0000","last_undegraded":"2023-10-21T05:23:45.719938+0000","last_fullsized":"2023-10-21T05:23:45.719938+0000","mapping_epoch":13,"log_start":"80'8100","ondisk_log_start":"80'8100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":273,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:00.033081+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:21:00.033081+0000","last_peered":"2023-10-21T05:21:00.033081+0000","last_clean":"2023-10-21T05:21:00.033081+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:21:00.033081+0000","last_undegraded":"2023-10-21T05:21:00.033081+0000","last_fullsized":"2023-10-21T05:21:00.033081+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27739529235,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2076122,"num_read_kb":8180108,"num_write":2074507,"num_write_kb":8298022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16994775040,"data_stored":19888746534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40211,"ondisk_log_size":40211,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38803,"ondisk_log_size":38803,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739358,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10412744,"kb_used_data":8299072,"kb_used_omap":940,"kb_used_meta":2112723,"kb_avail":83959096,"statfs":{"total":96636764160,"available":85974114304,"internally_reserved":0,"allocated":8498249728,"data_stored":9944692681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":962719,"internal_metadata":2163429217},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:11 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.64400000000000002,"15min":0.58699999999999997},"min":{"1min":0.19600000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":3.4889999999999999,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.58499999999999996,"15min":0.56799999999999995},"min":{"1min":0.22700000000000001,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":4.4880000000000004,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.25800000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:23:11 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.625,"15min":0.59699999999999998},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":3.855,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.63200000000000001,"15min":0.57099999999999995},"min":{"1min":0.247,"5min":0.221,"15min":0.19800000000000001},"max":{"1min":0.54200000000000004,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.437}]}]},{"osd":1,"up_from":8,"seq":34359739356,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6121952,"kb_used_data":4201496,"kb_used_omap":478,"kb_used_meta":1919969,"kb_avail":88249888,"statfs":{"total":96636764160,"available":90367885312,"internally_reserved":0,"allocated":4302331904,"data_stored":5048205339,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1966048370},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,10,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:12 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51100000000000001,"15min":0.52400000000000002},"min":{"1min":0.23400000000000001,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":0.84899999999999998,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.51600000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.88700000000000001,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.33900000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:23:12 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.36099999999999999,"15min":0.41499999999999998},"min":{"1min":0.224,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":0.59299999999999997,"5min":0.79200000000000004,"15min":8.7669999999999995},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.378,"15min":0.38800000000000001},"min":{"1min":0.23499999999999999,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":0.55600000000000005,"5min":1.468,"15min":4.1920000000000002},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739359,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5655704,"kb_used_data":4097488,"kb_used_omap":504,"kb_used_meta":1557703,"kb_avail":88716136,"statfs":{"total":96636764160,"available":90845323264,"internally_reserved":0,"allocated":4195827712,"data_stored":4896689132,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1595088249},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,11,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:23:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":0.56399999999999995,"15min":0.495},"min":{"1min":0.22800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":6.0030000000000001,"5min":6.0030000000000001,"15min":15.727},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.5409999999999999,"5min":0.69799999999999995,"15min":0.57599999999999996},"min":{"1min":0.29399999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":6.0529999999999999,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.221}]},{"osd":2,"last update":"Sat Oct 21 05:23:21 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":0.69799999999999995,"15min":0.56899999999999995},"min":{"1min":0.27600000000000002,"5min":0.22,"15min":0.22},"max":{"1min":5.3799999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":0.78800000000000003,"15min":0.66300000000000003},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":5.3200000000000003,"5min":9.8680000000000003,"15min":10.967000000000001},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4195565568,"data_stored":4896501760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4301619200,"data_stored":5047668755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8497590272,"data_stored":9944576019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:23:50.064 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:23:50.064 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:23:52.071 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3252b0 2023-10-21T05:23:52.071 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:23:52.072 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:23:52.072 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:23:52.072 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:23:52.073 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T05:23:52.073 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:23:52.073 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:23:52.073 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:23:52.074 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:23:52.074 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:23:52.074 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:23:52.075 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:23:52.075 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:23:52.062+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:24:10.065 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:24:10.686 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:24:10.687 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:24:10.711 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2575,"stamp":"2023-10-21T05:24:10.273235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27765181569,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2102474,"num_read_kb":8225341,"num_write":2082546,"num_write_kb":8329427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79362,"ondisk_log_size":79362,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21855720,"kb_used_data":16656012,"kb_used_omap":1925,"kb_used_meta":5197754,"kb_avail":261259800,"statfs":{"total":289910292480,"available":267530035200,"internally_reserved":0,"allocated":17055756288,"data_stored":19948934096,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1971524,"internal_metadata":5322500796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,7,33,2,3,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12705792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3687,"num_read_kb":14676,"num_write":3722,"num_write_kb":14888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005042"},"pg_stats":[{"pgid":"2.7","version":"96'14569","reported_seq":19933,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:04.553244+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:24:04.553244+0000","last_peered":"2023-10-21T05:24:04.553244+0000","last_clean":"2023-10-21T05:24:04.553244+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:24:04.553244+0000","last_undegraded":"2023-10-21T05:24:04.553244+0000","last_fullsized":"2023-10-21T05:24:04.553244+0000","mapping_epoch":13,"log_start":"80'4500","ondisk_log_start":"80'4500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7430","reported_seq":11156,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:04.551992+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:24:04.551992+0000","last_peered":"2023-10-21T05:24:04.551992+0000","last_clean":"2023-10-21T05:24:04.551992+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:24:04.551992+0000","last_undegraded":"2023-10-21T05:24:04.551992+0000","last_fullsized":"2023-10-21T05:24:04.551992+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3724","reported_seq":5232,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:04.426504+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:24:04.426504+0000","last_peered":"2023-10-21T05:24:04.426504+0000","last_clean":"2023-10-21T05:24:04.426504+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:24:04.426504+0000","last_undegraded":"2023-10-21T05:24:04.426504+0000","last_fullsized":"2023-10-21T05:24:04.426504+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3724,"log_dups_size":0,"ondisk_log_size":3724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7376","reported_seq":10512,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:04.552390+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:24:04.552390+0000","last_peered":"2023-10-21T05:24:04.552390+0000","last_clean":"2023-10-21T05:24:04.552390+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:24:04.552390+0000","last_undegraded":"2023-10-21T05:24:04.552390+0000","last_fullsized":"2023-10-21T05:24:04.552390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7376,"log_dups_size":0,"ondisk_log_size":7376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'532499","reported_seq":1058959,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:05.306925+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:24:05.306925+0000","last_peered":"2023-10-21T05:24:05.306925+0000","last_clean":"2023-10-21T05:24:05.306925+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:24:05.306925+0000","last_undegraded":"2023-10-21T05:24:05.306925+0000","last_fullsized":"2023-10-21T05:24:05.306925+0000","mapping_epoch":16,"log_start":"96'522418","ondisk_log_start":"96'522418","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7032258560,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":527508,"num_read_kb":2078784,"num_write":529008,"num_write_kb":2116032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":228,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:14.343638+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:21:14.343638+0000","last_peered":"2023-10-21T05:21:14.343638+0000","last_clean":"2023-10-21T05:21:14.343638+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:21:14.343638+0000","last_undegraded":"2023-10-21T05:21:14.343638+0000","last_fullsized":"2023-10-21T05:21:14.343638+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'536995","reported_seq":1070846,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:07.153966+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:24:07.153966+0000","last_peered":"2023-10-21T05:24:07.153966+0000","last_clean":"2023-10-21T05:24:07.153966+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:24:07.153966+0000","last_undegraded":"2023-10-21T05:24:07.153966+0000","last_fullsized":"2023-10-21T05:24:07.153966+0000","mapping_epoch":16,"log_start":"96'526919","ondisk_log_start":"96'526919","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7154040832,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":534968,"num_read_kb":2107980,"num_write":533359,"num_write_kb":2133436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4172,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:04.562342+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:24:04.562342+0000","last_peered":"2023-10-21T05:24:04.562342+0000","last_clean":"2023-10-21T05:24:04.562342+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:24:04.562342+0000","last_undegraded":"2023-10-21T05:24:04.562342+0000","last_fullsized":"2023-10-21T05:24:04.562342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'502234","reported_seq":1002112,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:05.299236+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:24:05.299236+0000","last_peered":"2023-10-21T05:24:05.299236+0000","last_clean":"2023-10-21T05:24:05.299236+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:24:05.299236+0000","last_undegraded":"2023-10-21T05:24:05.299236+0000","last_fullsized":"2023-10-21T05:24:05.299236+0000","mapping_epoch":16,"log_start":"96'492220","ondisk_log_start":"96'492220","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":870,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 202s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6621659136,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":500686,"num_read_kb":1972788,"num_write":498949,"num_write_kb":1995796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4108,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:04.552939+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:24:04.552939+0000","last_peered":"2023-10-21T05:24:04.552939+0000","last_clean":"2023-10-21T05:24:04.552939+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:24:04.552939+0000","last_undegraded":"2023-10-21T05:24:04.552939+0000","last_fullsized":"2023-10-21T05:24:04.552939+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'523220","reported_seq":1042381,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:07.179479+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:24:07.179479+0000","last_peered":"2023-10-21T05:24:07.179479+0000","last_clean":"2023-10-21T05:24:07.179479+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:24:07.179479+0000","last_undegraded":"2023-10-21T05:24:07.179479+0000","last_fullsized":"2023-10-21T05:24:07.179479+0000","mapping_epoch":16,"log_start":"96'513126","ondisk_log_start":"96'513126","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1660,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 453s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6956630035,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":520224,"num_read_kb":2049468,"num_write":520492,"num_write_kb":2081962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18181","reported_seq":24978,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:04.554864+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:24:04.554864+0000","last_peered":"2023-10-21T05:24:04.554864+0000","last_clean":"2023-10-21T05:24:04.554864+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:24:04.554864+0000","last_undegraded":"2023-10-21T05:24:04.554864+0000","last_fullsized":"2023-10-21T05:24:04.554864+0000","mapping_epoch":13,"log_start":"80'8100","ondisk_log_start":"80'8100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":274,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.384655+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:22:01.384655+0000","last_peered":"2023-10-21T05:22:01.384655+0000","last_clean":"2023-10-21T05:22:01.384655+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:22:01.384655+0000","last_undegraded":"2023-10-21T05:22:01.384655+0000","last_fullsized":"2023-10-21T05:22:01.384655+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27764588563,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2083386,"num_read_kb":8209020,"num_write":2081808,"num_write_kb":8327226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17054662656,"data_stored":19948634150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40265,"ondisk_log_size":40265,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38998,"ondisk_log_size":38998,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739362,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9958544,"kb_used_data":8327748,"kb_used_omap":941,"kb_used_meta":1629842,"kb_avail":84413296,"statfs":{"total":96636764160,"available":86439215104,"internally_reserved":0,"allocated":8527613952,"data_stored":9974056905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964399,"internal_metadata":1668958417},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:11 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.64400000000000002,"15min":0.58699999999999997},"min":{"1min":0.19600000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":3.4889999999999999,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.58499999999999996,"15min":0.56799999999999995},"min":{"1min":0.22700000000000001,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":4.4880000000000004,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.27100000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:23:11 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.625,"15min":0.59699999999999998},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":3.855,"5min":7.5099999999999998,"15min":7.8449999999999998},"last":0.311},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.63200000000000001,"15min":0.57099999999999995},"min":{"1min":0.247,"5min":0.221,"15min":0.19800000000000001},"max":{"1min":0.54200000000000004,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.22700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739360,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6172604,"kb_used_data":4216052,"kb_used_omap":478,"kb_used_meta":1956065,"kb_avail":88199236,"statfs":{"total":96636764160,"available":90316017664,"internally_reserved":0,"allocated":4317237248,"data_stored":5063110683,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":2003010674},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:23:12 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51100000000000001,"15min":0.52400000000000002},"min":{"1min":0.23400000000000001,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":0.84899999999999998,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.51600000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.88700000000000001,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.64400000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:23:12 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.36099999999999999,"15min":0.41499999999999998},"min":{"1min":0.224,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":0.59299999999999997,"5min":0.79200000000000004,"15min":8.7669999999999995},"last":0.31},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.378,"15min":0.38800000000000001},"min":{"1min":0.23499999999999999,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":0.55600000000000005,"5min":1.468,"15min":4.1920000000000002},"last":0.34899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739363,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5724572,"kb_used_data":4112212,"kb_used_omap":504,"kb_used_meta":1611847,"kb_avail":88647268,"statfs":{"total":96636764160,"available":90774802432,"internally_reserved":0,"allocated":4210905088,"data_stored":4911766508,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1650531705},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,31,1,3,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:23:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":0.56399999999999995,"15min":0.495},"min":{"1min":0.22800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":6.0030000000000001,"5min":6.0030000000000001,"15min":15.727},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.5409999999999999,"5min":0.69799999999999995,"15min":0.57599999999999996},"min":{"1min":0.29399999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":6.0529999999999999,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.435}]},{"osd":2,"last update":"Sat Oct 21 05:23:21 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":0.69799999999999995,"15min":0.56899999999999995},"min":{"1min":0.27600000000000002,"5min":0.22,"15min":0.22},"max":{"1min":5.3799999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.2509999999999999,"5min":0.78800000000000003,"15min":0.66300000000000003},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":5.3200000000000003,"5min":9.8680000000000003,"15min":10.967000000000001},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4210511872,"data_stored":4911448064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4316577792,"data_stored":5062627347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8527572992,"data_stored":9974558739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:24:10.713 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:24:10.714 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:24:30.715 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:24:31.860 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:24:31.861 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:24:31.909 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2585,"stamp":"2023-10-21T05:24:30.281042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27790433409,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2109857,"num_read_kb":8254749,"num_write":2089832,"num_write_kb":8358571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79265,"ondisk_log_size":79265,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21394580,"kb_used_data":16713976,"kb_used_omap":1925,"kb_used_meta":4678650,"kb_avail":261720940,"statfs":{"total":289910292480,"available":268002242560,"internally_reserved":0,"allocated":17115111424,"data_stored":20008289232,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1971524,"internal_metadata":4790938300},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,5,6,5,14,9,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12193792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3495,"num_read_kb":13908,"num_write":3517,"num_write_kb":14068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006105"},"pg_stats":[{"pgid":"2.7","version":"96'14621","reported_seq":20002,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:26.659146+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:24:26.659146+0000","last_peered":"2023-10-21T05:24:26.659146+0000","last_clean":"2023-10-21T05:24:26.659146+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:24:26.659146+0000","last_undegraded":"2023-10-21T05:24:26.659146+0000","last_fullsized":"2023-10-21T05:24:26.659146+0000","mapping_epoch":13,"log_start":"80'4600","ondisk_log_start":"80'4600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7456","reported_seq":11191,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:26.647034+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:24:26.647034+0000","last_peered":"2023-10-21T05:24:26.647034+0000","last_clean":"2023-10-21T05:24:26.647034+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:24:26.647034+0000","last_undegraded":"2023-10-21T05:24:26.647034+0000","last_fullsized":"2023-10-21T05:24:26.647034+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7456,"log_dups_size":0,"ondisk_log_size":7456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3737","reported_seq":5249,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:25.032613+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:24:25.032613+0000","last_peered":"2023-10-21T05:24:25.032613+0000","last_clean":"2023-10-21T05:24:25.032613+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:24:25.032613+0000","last_undegraded":"2023-10-21T05:24:25.032613+0000","last_fullsized":"2023-10-21T05:24:25.032613+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3737,"log_dups_size":0,"ondisk_log_size":3737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7402","reported_seq":10546,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:26.631772+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:24:26.631772+0000","last_peered":"2023-10-21T05:24:26.631772+0000","last_clean":"2023-10-21T05:24:26.631772+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:24:26.631772+0000","last_undegraded":"2023-10-21T05:24:26.631772+0000","last_fullsized":"2023-10-21T05:24:26.631772+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7402,"log_dups_size":0,"ondisk_log_size":7402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'534366","reported_seq":1062689,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:25.568741+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:24:25.568741+0000","last_peered":"2023-10-21T05:24:25.568741+0000","last_clean":"2023-10-21T05:24:25.568741+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:24:25.568741+0000","last_undegraded":"2023-10-21T05:24:25.568741+0000","last_fullsized":"2023-10-21T05:24:25.568741+0000","mapping_epoch":16,"log_start":"96'524318","ondisk_log_start":"96'524318","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7038697472,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":529374,"num_read_kb":2086224,"num_write":530875,"num_write_kb":2123500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":228,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:14.343638+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:21:14.343638+0000","last_peered":"2023-10-21T05:21:14.343638+0000","last_clean":"2023-10-21T05:21:14.343638+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:21:14.343638+0000","last_undegraded":"2023-10-21T05:21:14.343638+0000","last_fullsized":"2023-10-21T05:21:14.343638+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'538965","reported_seq":1074754,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:27.312933+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:24:27.312933+0000","last_peered":"2023-10-21T05:24:27.312933+0000","last_clean":"2023-10-21T05:24:27.312933+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:24:27.312933+0000","last_undegraded":"2023-10-21T05:24:27.312933+0000","last_fullsized":"2023-10-21T05:24:27.312933+0000","mapping_epoch":16,"log_start":"96'528919","ondisk_log_start":"96'528919","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7160975360,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":536911,"num_read_kb":2115688,"num_write":535329,"num_write_kb":2141316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4184,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:25.035163+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:24:25.035163+0000","last_peered":"2023-10-21T05:24:25.035163+0000","last_clean":"2023-10-21T05:24:25.035163+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:24:25.035163+0000","last_undegraded":"2023-10-21T05:24:25.035163+0000","last_fullsized":"2023-10-21T05:24:25.035163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'503889","reported_seq":1005558,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:25.538167+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:24:25.538167+0000","last_peered":"2023-10-21T05:24:25.538167+0000","last_clean":"2023-10-21T05:24:25.538167+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:24:25.538167+0000","last_undegraded":"2023-10-21T05:24:25.538167+0000","last_fullsized":"2023-10-21T05:24:25.538167+0000","mapping_epoch":16,"log_start":"96'493820","ondisk_log_start":"96'493820","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":953,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 222s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6627233792,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":502472,"num_read_kb":1979920,"num_write":500598,"num_write_kb":2002392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4120,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:25.058885+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:24:25.058885+0000","last_peered":"2023-10-21T05:24:25.058885+0000","last_clean":"2023-10-21T05:24:25.058885+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:24:25.058885+0000","last_undegraded":"2023-10-21T05:24:25.058885+0000","last_fullsized":"2023-10-21T05:24:25.058885+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'525049","reported_seq":1046005,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:27.329741+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:24:27.329741+0000","last_peered":"2023-10-21T05:24:27.329741+0000","last_clean":"2023-10-21T05:24:27.329741+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:24:27.329741+0000","last_undegraded":"2023-10-21T05:24:27.329741+0000","last_fullsized":"2023-10-21T05:24:27.329741+0000","mapping_epoch":16,"log_start":"96'515026","ondisk_log_start":"96'515026","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1709,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 473s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6962933779,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":522012,"num_read_kb":2056596,"num_write":522292,"num_write_kb":2089162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18246","reported_seq":25063,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:26.714702+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:24:26.714702+0000","last_peered":"2023-10-21T05:24:26.714702+0000","last_clean":"2023-10-21T05:24:26.714702+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:24:26.714702+0000","last_undegraded":"2023-10-21T05:24:26.714702+0000","last_fullsized":"2023-10-21T05:24:26.714702+0000","mapping_epoch":13,"log_start":"80'8200","ondisk_log_start":"80'8200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":274,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.384655+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:22:01.384655+0000","last_peered":"2023-10-21T05:22:01.384655+0000","last_clean":"2023-10-21T05:22:01.384655+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:22:01.384655+0000","last_undegraded":"2023-10-21T05:22:01.384655+0000","last_fullsized":"2023-10-21T05:22:01.384655+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27789840403,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2090769,"num_read_kb":8238428,"num_write":2089094,"num_write_kb":8356370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17114173440,"data_stored":20008144934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40186,"ondisk_log_size":40186,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":38980,"ondisk_log_size":38980,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739366,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10013828,"kb_used_data":8356472,"kb_used_omap":941,"kb_used_meta":1656402,"kb_avail":84358012,"statfs":{"total":96636764160,"available":86382604288,"internally_reserved":0,"allocated":8557027328,"data_stored":10003470281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964399,"internal_metadata":1696155857},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:12 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.69299999999999995,"15min":0.61099999999999999},"min":{"1min":0.222,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":11.939,"5min":11.939,"15min":11.939},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.68100000000000005,"15min":0.60199999999999998},"min":{"1min":0.223,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":7.6029999999999998,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.28299999999999997}]},{"osd":1,"last update":"Sat Oct 21 05:24:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.65800000000000003,"15min":0.622},"min":{"1min":0.21099999999999999,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":11.901999999999999,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.55400000000000005,"15min":0.56499999999999995},"min":{"1min":0.22700000000000001,"5min":0.221,"15min":0.19800000000000001},"max":{"1min":2.1299999999999999,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.44}]}]},{"osd":1,"up_from":8,"seq":34359739364,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5609844,"kb_used_data":4230892,"kb_used_omap":478,"kb_used_meta":1378465,"kb_avail":88761996,"statfs":{"total":96636764160,"available":90892283904,"internally_reserved":0,"allocated":4332433408,"data_stored":5078306843,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1411548274},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,3,1,5,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.57999999999999996,"15min":0.52800000000000002},"min":{"1min":0.23400000000000001,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":5.7809999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.58499999999999996,"15min":0.52200000000000002},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":2.625,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.314}]},{"osd":2,"last update":"Sat Oct 21 05:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.38500000000000001,"15min":0.41699999999999998},"min":{"1min":0.223,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":2.4889999999999999,"5min":2.4889999999999999,"15min":8.7669999999999995},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.44900000000000001,"15min":0.40999999999999998},"min":{"1min":0.28699999999999998,"5min":0.223,"15min":0.221},"max":{"1min":2.5870000000000002,"5min":2.5870000000000002,"15min":4.1920000000000002},"last":0.441}]}]},{"osd":0,"up_from":8,"seq":34359739367,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5770908,"kb_used_data":4126612,"kb_used_omap":504,"kb_used_meta":1643783,"kb_avail":88600932,"statfs":{"total":96636764160,"available":90727354368,"internally_reserved":0,"allocated":4225650688,"data_stored":4926512108,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1683234169},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,2,4,4,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.626,"15min":0.51600000000000001},"min":{"1min":0.19700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.2220000000000004,"5min":6.0030000000000001,"15min":15.727},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.379,"5min":0.69299999999999995,"15min":0.57399999999999995},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53200000000000003,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.53200000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.67200000000000004,"15min":0.56899999999999995},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.60999999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.80600000000000005,"15min":0.66900000000000004},"min":{"1min":0.29499999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.2360000000000002,"5min":9.8680000000000003,"15min":10.967000000000001},"last":0.58899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4224917504,"data_stored":4925853696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4332023808,"data_stored":5078073363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8557232128,"data_stored":10004217875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:24:31.911 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:24:31.912 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:24:49.282 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.274+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:24:49.292 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.282+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:24:49.292 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.282+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:24:49.292 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.282+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:24:49.293 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.282+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b8831e0 2023-10-21T05:24:49.293 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.282+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:24:49.297 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.290+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:24:49.298 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.290+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:24:49.298 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.290+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:24:49.298 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.290+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf42750 2023-10-21T05:24:49.299 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.290+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:24:49.299 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.290+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:24:49.299 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:24:49.290+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:24:51.913 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:24:53.014 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:24:53.015 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:24:53.053 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2596,"stamp":"2023-10-21T05:24:52.284731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27818916993,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2118065,"num_read_kb":8287453,"num_write":2098094,"num_write_kb":8391619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79300,"ondisk_log_size":79300,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21650088,"kb_used_data":16784844,"kb_used_omap":1925,"kb_used_meta":4863290,"kb_avail":261465432,"statfs":{"total":289910292480,"available":267740602368,"internally_reserved":0,"allocated":17187680256,"data_stored":20080858064,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1971524,"internal_metadata":4980009660},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,4,15,7,13,7,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15609856,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4573,"num_read_kb":18228,"num_write":4542,"num_write_kb":18168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002012"},"pg_stats":[{"pgid":"2.7","version":"96'14681","reported_seq":20078,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:46.172880+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:24:46.172880+0000","last_peered":"2023-10-21T05:24:46.172880+0000","last_clean":"2023-10-21T05:24:46.172880+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:24:46.172880+0000","last_undegraded":"2023-10-21T05:24:46.172880+0000","last_fullsized":"2023-10-21T05:24:46.172880+0000","mapping_epoch":13,"log_start":"80'4600","ondisk_log_start":"80'4600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7486","reported_seq":11229,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:46.149977+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:24:46.149977+0000","last_peered":"2023-10-21T05:24:46.149977+0000","last_clean":"2023-10-21T05:24:46.149977+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:24:46.149977+0000","last_undegraded":"2023-10-21T05:24:46.149977+0000","last_fullsized":"2023-10-21T05:24:46.149977+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7486,"log_dups_size":0,"ondisk_log_size":7486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3752","reported_seq":5268,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:45.963426+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:24:45.963426+0000","last_peered":"2023-10-21T05:24:45.963426+0000","last_clean":"2023-10-21T05:24:45.963426+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:24:45.963426+0000","last_undegraded":"2023-10-21T05:24:45.963426+0000","last_fullsized":"2023-10-21T05:24:45.963426+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3752,"log_dups_size":0,"ondisk_log_size":3752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7432","reported_seq":10584,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:46.156932+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:24:46.156932+0000","last_peered":"2023-10-21T05:24:46.156932+0000","last_clean":"2023-10-21T05:24:46.156932+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:24:46.156932+0000","last_undegraded":"2023-10-21T05:24:46.156932+0000","last_fullsized":"2023-10-21T05:24:46.156932+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7432,"log_dups_size":0,"ondisk_log_size":7432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'536748","reported_seq":1067352,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:50.705865+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:24:50.705865+0000","last_peered":"2023-10-21T05:24:50.705865+0000","last_clean":"2023-10-21T05:24:50.705865+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:24:50.705865+0000","last_undegraded":"2023-10-21T05:24:50.705865+0000","last_fullsized":"2023-10-21T05:24:50.705865+0000","mapping_epoch":16,"log_start":"96'526718","ondisk_log_start":"96'526718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7046959104,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":531650,"num_read_kb":2095292,"num_write":533257,"num_write_kb":2133028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":228,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:14.343638+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:21:14.343638+0000","last_peered":"2023-10-21T05:21:14.343638+0000","last_clean":"2023-10-21T05:21:14.343638+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:21:14.343638+0000","last_undegraded":"2023-10-21T05:21:14.343638+0000","last_fullsized":"2023-10-21T05:21:14.343638+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'540842","reported_seq":1078571,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:47.437164+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:24:47.437164+0000","last_peered":"2023-10-21T05:24:47.437164+0000","last_clean":"2023-10-21T05:24:47.437164+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:24:47.437164+0000","last_undegraded":"2023-10-21T05:24:47.437164+0000","last_fullsized":"2023-10-21T05:24:47.437164+0000","mapping_epoch":16,"log_start":"96'530819","ondisk_log_start":"96'530819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7167651840,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":538849,"num_read_kb":2123404,"num_write":537206,"num_write_kb":2148824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4199,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:50.632984+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:24:50.632984+0000","last_peered":"2023-10-21T05:24:50.632984+0000","last_clean":"2023-10-21T05:24:50.632984+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:24:50.632984+0000","last_undegraded":"2023-10-21T05:24:50.632984+0000","last_fullsized":"2023-10-21T05:24:50.632984+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'506026","reported_seq":1009803,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:50.724404+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:24:50.724404+0000","last_peered":"2023-10-21T05:24:50.724404+0000","last_clean":"2023-10-21T05:24:50.724404+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:24:50.724404+0000","last_undegraded":"2023-10-21T05:24:50.724404+0000","last_fullsized":"2023-10-21T05:24:50.724404+0000","mapping_epoch":16,"log_start":"96'496020","ondisk_log_start":"96'496020","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":1074,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 247s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6634369024,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":504565,"num_read_kb":1988260,"num_write":502729,"num_write_kb":2010916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4135,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:50.632224+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:24:50.632224+0000","last_peered":"2023-10-21T05:24:50.632224+0000","last_clean":"2023-10-21T05:24:50.632224+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:24:50.632224+0000","last_undegraded":"2023-10-21T05:24:50.632224+0000","last_fullsized":"2023-10-21T05:24:50.632224+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'526978","reported_seq":1049844,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:24:47.461819+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:24:47.461819+0000","last_peered":"2023-10-21T05:24:47.461819+0000","last_clean":"2023-10-21T05:24:47.461819+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:24:47.461819+0000","last_undegraded":"2023-10-21T05:24:47.461819+0000","last_fullsized":"2023-10-21T05:24:47.461819+0000","mapping_epoch":16,"log_start":"96'516926","ondisk_log_start":"96'516926","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1798,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 493s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6969344019,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":523913,"num_read_kb":2064176,"num_write":524164,"num_write_kb":2096650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18321","reported_seq":25158,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:46.188066+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:24:46.188066+0000","last_peered":"2023-10-21T05:24:46.188066+0000","last_clean":"2023-10-21T05:24:46.188066+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:24:46.188066+0000","last_undegraded":"2023-10-21T05:24:46.188066+0000","last_fullsized":"2023-10-21T05:24:46.188066+0000","mapping_epoch":13,"log_start":"80'8300","ondisk_log_start":"80'8300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":274,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.384655+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:22:01.384655+0000","last_peered":"2023-10-21T05:22:01.384655+0000","last_clean":"2023-10-21T05:22:01.384655+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:22:01.384655+0000","last_undegraded":"2023-10-21T05:22:01.384655+0000","last_fullsized":"2023-10-21T05:22:01.384655+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27818323987,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2098977,"num_read_kb":8271132,"num_write":2097356,"num_write_kb":8389418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17185685504,"data_stored":20079656998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40111,"ondisk_log_size":40111,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39090,"ondisk_log_size":39090,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10159976,"kb_used_data":8394332,"kb_used_omap":941,"kb_used_meta":1764690,"kb_avail":84211864,"statfs":{"total":96636764160,"available":86232948736,"internally_reserved":0,"allocated":8595795968,"data_stored":10042238921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964399,"internal_metadata":1807042769},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:12 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.69299999999999995,"15min":0.61099999999999999},"min":{"1min":0.222,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":11.939,"5min":11.939,"15min":11.939},"last":2.2360000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.68100000000000005,"15min":0.60199999999999998},"min":{"1min":0.223,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":7.6029999999999998,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":2.214}]},{"osd":1,"last update":"Sat Oct 21 05:24:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.65800000000000003,"15min":0.622},"min":{"1min":0.21099999999999999,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":11.901999999999999,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.55400000000000005,"15min":0.56499999999999995},"min":{"1min":0.22700000000000001,"5min":0.221,"15min":0.19800000000000001},"max":{"1min":2.1299999999999999,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.27800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739368,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5660900,"kb_used_data":4245852,"kb_used_omap":478,"kb_used_meta":1414561,"kb_avail":88710940,"statfs":{"total":96636764160,"available":90840002560,"internally_reserved":0,"allocated":4347752448,"data_stored":5093625883,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1448510578},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,7,4,7,7,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.57999999999999996,"15min":0.52800000000000002},"min":{"1min":0.23400000000000001,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":5.7809999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.58499999999999996,"15min":0.52200000000000002},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":2.625,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.47499999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.38500000000000001,"15min":0.41699999999999998},"min":{"1min":0.223,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":2.4889999999999999,"5min":2.4889999999999999,"15min":8.7669999999999995},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.44900000000000001,"15min":0.40999999999999998},"min":{"1min":0.28699999999999998,"5min":0.223,"15min":0.221},"max":{"1min":2.5870000000000002,"5min":2.5870000000000002,"15min":4.1920000000000002},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739372,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5829212,"kb_used_data":4144660,"kb_used_omap":504,"kb_used_meta":1684039,"kb_avail":88542628,"statfs":{"total":96636764160,"available":90667651072,"internally_reserved":0,"allocated":4244131840,"data_stored":4944993260,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1724456313},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.626,"15min":0.51600000000000001},"min":{"1min":0.19700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.2220000000000004,"5min":6.0030000000000001,"15min":15.727},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.69299999999999995,"15min":0.57399999999999995},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53200000000000003,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.39500000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.67200000000000004,"15min":0.56899999999999995},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.60999999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.80600000000000005,"15min":0.66900000000000004},"min":{"1min":0.29499999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.2360000000000002,"5min":9.8680000000000003,"15min":10.967000000000001},"last":0.48199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4243419136,"data_stored":4944355328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4347367424,"data_stored":5093416979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8594898944,"data_stored":10041884691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:24:53.055 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:24:53.055 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:25:09.246 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:25:09.238+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:25:09.257 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:25:09.250+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:25:09.269 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:25:09.258+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:25:09.270 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:25:09.258+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:25:13.057 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:25:13.766 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:25:13.766 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:25:13.804 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2606,"stamp":"2023-10-21T05:25:12.291914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27844713601,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2125649,"num_read_kb":8317669,"num_write":2105540,"num_write_kb":8421403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79353,"ondisk_log_size":79353,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21868208,"kb_used_data":16844696,"kb_used_omap":1925,"kb_used_meta":5021562,"kb_avail":261247312,"statfs":{"total":289910292480,"available":267517247488,"internally_reserved":0,"allocated":17248968704,"data_stored":20142146512,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1971524,"internal_metadata":5142080188},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,5,15,18,7,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16031744,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4715,"num_read_kb":18788,"num_write":4643,"num_write_kb":18572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003939"},"pg_stats":[{"pgid":"2.7","version":"96'14729","reported_seq":20142,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:06.122471+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:25:06.122471+0000","last_peered":"2023-10-21T05:25:06.122471+0000","last_clean":"2023-10-21T05:25:06.122471+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:25:06.122471+0000","last_undegraded":"2023-10-21T05:25:06.122471+0000","last_fullsized":"2023-10-21T05:25:06.122471+0000","mapping_epoch":13,"log_start":"80'4700","ondisk_log_start":"80'4700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7510","reported_seq":11261,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:06.105970+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:25:06.105970+0000","last_peered":"2023-10-21T05:25:06.105970+0000","last_clean":"2023-10-21T05:25:06.105970+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:25:06.105970+0000","last_undegraded":"2023-10-21T05:25:06.105970+0000","last_fullsized":"2023-10-21T05:25:06.105970+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7510,"log_dups_size":0,"ondisk_log_size":7510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3764","reported_seq":5284,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:05.992176+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:25:05.992176+0000","last_peered":"2023-10-21T05:25:05.992176+0000","last_clean":"2023-10-21T05:25:05.992176+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:25:05.992176+0000","last_undegraded":"2023-10-21T05:25:05.992176+0000","last_fullsized":"2023-10-21T05:25:05.992176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3764,"log_dups_size":0,"ondisk_log_size":3764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7456","reported_seq":10616,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:06.105944+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:25:06.105944+0000","last_peered":"2023-10-21T05:25:06.105944+0000","last_clean":"2023-10-21T05:25:06.105944+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:25:06.105944+0000","last_undegraded":"2023-10-21T05:25:06.105944+0000","last_fullsized":"2023-10-21T05:25:06.105944+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7456,"log_dups_size":0,"ondisk_log_size":7456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'538630","reported_seq":1071132,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:10.873086+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:25:10.873086+0000","last_peered":"2023-10-21T05:25:10.873086+0000","last_clean":"2023-10-21T05:25:10.873086+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:25:10.873086+0000","last_undegraded":"2023-10-21T05:25:10.873086+0000","last_fullsized":"2023-10-21T05:25:10.873086+0000","mapping_epoch":16,"log_start":"96'528618","ondisk_log_start":"96'528618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7053393920,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":533551,"num_read_kb":2102872,"num_write":535139,"num_write_kb":2140556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":228,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:21:14.343638+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:21:14.343638+0000","last_peered":"2023-10-21T05:21:14.343638+0000","last_clean":"2023-10-21T05:21:14.343638+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:21:14.343638+0000","last_undegraded":"2023-10-21T05:21:14.343638+0000","last_fullsized":"2023-10-21T05:21:14.343638+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'542754","reported_seq":1082494,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:07.585589+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:25:07.585589+0000","last_peered":"2023-10-21T05:25:07.585589+0000","last_clean":"2023-10-21T05:25:07.585589+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:25:07.585589+0000","last_undegraded":"2023-10-21T05:25:07.585589+0000","last_fullsized":"2023-10-21T05:25:07.585589+0000","mapping_epoch":16,"log_start":"96'532719","ondisk_log_start":"96'532719","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7174385664,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":540857,"num_read_kb":2131408,"num_write":539118,"num_write_kb":2156472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4212,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:11.119310+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:25:11.119310+0000","last_peered":"2023-10-21T05:25:11.119310+0000","last_clean":"2023-10-21T05:25:11.119310+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:25:11.119310+0000","last_undegraded":"2023-10-21T05:25:11.119310+0000","last_fullsized":"2023-10-21T05:25:11.119310+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'507828","reported_seq":1013402,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:25:10.886420+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:25:10.886420+0000","last_peered":"2023-10-21T05:25:10.886420+0000","last_clean":"2023-10-21T05:25:10.886420+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:25:10.886420+0000","last_undegraded":"2023-10-21T05:25:10.886420+0000","last_fullsized":"2023-10-21T05:25:10.886420+0000","mapping_epoch":16,"log_start":"96'497820","ondisk_log_start":"96'497820","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":1174,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 267s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6640513024,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":506347,"num_read_kb":1995340,"num_write":504529,"num_write_kb":2018116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4147,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:11.206657+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:25:11.206657+0000","last_peered":"2023-10-21T05:25:11.206657+0000","last_clean":"2023-10-21T05:25:11.206657+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:25:11.206657+0000","last_undegraded":"2023-10-21T05:25:11.206657+0000","last_fullsized":"2023-10-21T05:25:11.206657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'528867","reported_seq":1053638,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:25:07.594839+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:25:07.594839+0000","last_peered":"2023-10-21T05:25:07.594839+0000","last_clean":"2023-10-21T05:25:07.594839+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:25:07.594839+0000","last_undegraded":"2023-10-21T05:25:07.594839+0000","last_fullsized":"2023-10-21T05:25:07.594839+0000","mapping_epoch":16,"log_start":"96'518826","ondisk_log_start":"96'518826","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":1859,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 514s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6975827987,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":525806,"num_read_kb":2071728,"num_write":526016,"num_write_kb":2104058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18381","reported_seq":25238,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:06.138866+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:25:06.138866+0000","last_peered":"2023-10-21T05:25:06.138866+0000","last_clean":"2023-10-21T05:25:06.138866+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:25:06.138866+0000","last_undegraded":"2023-10-21T05:25:06.138866+0000","last_fullsized":"2023-10-21T05:25:06.138866+0000","mapping_epoch":13,"log_start":"80'8300","ondisk_log_start":"80'8300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":274,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.384655+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:22:01.384655+0000","last_peered":"2023-10-21T05:22:01.384655+0000","last_clean":"2023-10-21T05:22:01.384655+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:22:01.384655+0000","last_undegraded":"2023-10-21T05:22:01.384655+0000","last_fullsized":"2023-10-21T05:22:01.384655+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27844120595,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2106561,"num_read_kb":8301348,"num_write":2104802,"num_write_kb":8419202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17246728192,"data_stored":20140699686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40096,"ondisk_log_size":40096,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39158,"ondisk_log_size":39158,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739375,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10257476,"kb_used_data":8423804,"kb_used_omap":941,"kb_used_meta":1832722,"kb_avail":84114364,"statfs":{"total":96636764160,"available":86133108736,"internally_reserved":0,"allocated":8625975296,"data_stored":10072418249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964399,"internal_metadata":1876707537},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:12 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.69299999999999995,"15min":0.61099999999999999},"min":{"1min":0.222,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":11.939,"5min":11.939,"15min":11.939},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.68100000000000005,"15min":0.60199999999999998},"min":{"1min":0.223,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":7.6029999999999998,"5min":8.2439999999999998,"15min":8.8510000000000009},"last":0.28799999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:24:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.65800000000000003,"15min":0.622},"min":{"1min":0.21099999999999999,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":11.901999999999999,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.55400000000000005,"15min":0.56499999999999995},"min":{"1min":0.22700000000000001,"5min":0.221,"15min":0.19800000000000001},"max":{"1min":2.1299999999999999,"5min":8.4030000000000005,"15min":8.4030000000000005},"last":0.42999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739372,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5712688,"kb_used_data":4261544,"kb_used_omap":478,"kb_used_meta":1450657,"kb_avail":88659152,"statfs":{"total":96636764160,"available":90786971648,"internally_reserved":0,"allocated":4363821056,"data_stored":5109694491,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1485472882},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,5,10,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.57999999999999996,"15min":0.52800000000000002},"min":{"1min":0.23400000000000001,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":5.7809999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.58499999999999996,"15min":0.52200000000000002},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":2.625,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.28499999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:24:14 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.38500000000000001,"15min":0.41699999999999998},"min":{"1min":0.223,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":2.4889999999999999,"5min":2.4889999999999999,"15min":8.7669999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.44900000000000001,"15min":0.40999999999999998},"min":{"1min":0.28699999999999998,"5min":0.223,"15min":0.221},"max":{"1min":2.5870000000000002,"5min":2.5870000000000002,"15min":4.1920000000000002},"last":0.33300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739376,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5898044,"kb_used_data":4159348,"kb_used_omap":504,"kb_used_meta":1738183,"kb_avail":88473796,"statfs":{"total":96636764160,"available":90597167104,"internally_reserved":0,"allocated":4259172352,"data_stored":4960033772,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1779899769},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,7,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.626,"15min":0.51600000000000001},"min":{"1min":0.19700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.2220000000000004,"5min":6.0030000000000001,"15min":15.727},"last":0.315},{"interface":"front","average":{"1min":0.379,"5min":0.69299999999999995,"15min":0.57399999999999995},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53200000000000003,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.46300000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.67200000000000004,"15min":0.56899999999999995},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.60999999999999999,"5min":9.3109999999999999,"15min":9.3109999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.80600000000000005,"15min":0.66900000000000004},"min":{"1min":0.29499999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.2360000000000002,"5min":9.8680000000000003,"15min":10.967000000000001},"last":0.38200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4258480128,"data_stored":4959416320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4362792960,"data_stored":5108842515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8625455104,"data_stored":10072440851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:25:13.806 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:25:13.807 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:25:14.267 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:25:14.303 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:25:34.304 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:25:34.880 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:25:34.880 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:25:34.910 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2617,"stamp":"2023-10-21T05:25:34.295736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27873393793,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2134205,"num_read_kb":8351653,"num_write":2113746,"num_write_kb":8454227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79532,"ondisk_log_size":79532,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22132592,"kb_used_data":16906520,"kb_used_omap":1925,"kb_used_meta":5224122,"kb_avail":260982928,"statfs":{"total":289910292480,"available":267246518272,"internally_reserved":0,"allocated":17312276480,"data_stored":20205454288,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1971524,"internal_metadata":5349501628},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,4,5,29,16,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16039936,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4787,"num_read_kb":19008,"num_write":4602,"num_write_kb":18408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002117"},"pg_stats":[{"pgid":"2.7","version":"96'14781","reported_seq":20214,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:32.224381+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:25:32.224381+0000","last_peered":"2023-10-21T05:25:32.224381+0000","last_clean":"2023-10-21T05:25:32.224381+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:25:32.224381+0000","last_undegraded":"2023-10-21T05:25:32.224381+0000","last_fullsized":"2023-10-21T05:25:32.224381+0000","mapping_epoch":13,"log_start":"80'4700","ondisk_log_start":"80'4700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7536","reported_seq":11297,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:32.191349+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:25:32.191349+0000","last_peered":"2023-10-21T05:25:32.191349+0000","last_clean":"2023-10-21T05:25:32.191349+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:25:32.191349+0000","last_undegraded":"2023-10-21T05:25:32.191349+0000","last_fullsized":"2023-10-21T05:25:32.191349+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7536,"log_dups_size":0,"ondisk_log_size":7536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3777","reported_seq":5303,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:31.597446+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:25:31.597446+0000","last_peered":"2023-10-21T05:25:31.597446+0000","last_clean":"2023-10-21T05:25:31.597446+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:25:31.597446+0000","last_undegraded":"2023-10-21T05:25:31.597446+0000","last_fullsized":"2023-10-21T05:25:31.597446+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7482","reported_seq":10653,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:32.203256+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:25:32.203256+0000","last_peered":"2023-10-21T05:25:32.203256+0000","last_clean":"2023-10-21T05:25:32.203256+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:25:32.203256+0000","last_undegraded":"2023-10-21T05:25:32.203256+0000","last_fullsized":"2023-10-21T05:25:32.203256+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7482,"log_dups_size":0,"ondisk_log_size":7482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'540460","reported_seq":1074866,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:31.047797+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:25:31.047797+0000","last_peered":"2023-10-21T05:25:31.047797+0000","last_clean":"2023-10-21T05:25:31.047797+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:25:31.047797+0000","last_undegraded":"2023-10-21T05:25:31.047797+0000","last_fullsized":"2023-10-21T05:25:31.047797+0000","mapping_epoch":16,"log_start":"96'530418","ondisk_log_start":"96'530418","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7059755008,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":535454,"num_read_kb":2110432,"num_write":536969,"num_write_kb":2147876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":229,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:19.994954+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:23:19.994954+0000","last_peered":"2023-10-21T05:23:19.994954+0000","last_clean":"2023-10-21T05:23:19.994954+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:23:19.994954+0000","last_undegraded":"2023-10-21T05:23:19.994954+0000","last_fullsized":"2023-10-21T05:23:19.994954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'545072","reported_seq":1087220,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:32.764947+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:25:32.764947+0000","last_peered":"2023-10-21T05:25:32.764947+0000","last_clean":"2023-10-21T05:25:32.764947+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:25:32.764947+0000","last_undegraded":"2023-10-21T05:25:32.764947+0000","last_fullsized":"2023-10-21T05:25:32.764947+0000","mapping_epoch":16,"log_start":"96'535019","ondisk_log_start":"96'535019","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7182667776,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":543265,"num_read_kb":2140960,"num_write":541436,"num_write_kb":2165744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4224,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:31.603686+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:25:31.603686+0000","last_peered":"2023-10-21T05:25:31.603686+0000","last_clean":"2023-10-21T05:25:31.603686+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:25:31.603686+0000","last_undegraded":"2023-10-21T05:25:31.603686+0000","last_fullsized":"2023-10-21T05:25:31.603686+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'509600","reported_seq":1017102,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:25:31.059098+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:25:31.059098+0000","last_peered":"2023-10-21T05:25:31.059098+0000","last_clean":"2023-10-21T05:25:31.059098+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:25:31.059098+0000","last_undegraded":"2023-10-21T05:25:31.059098+0000","last_fullsized":"2023-10-21T05:25:31.059098+0000","mapping_epoch":16,"log_start":"96'499520","ondisk_log_start":"96'499520","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":1268,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 288s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6646562816,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":508263,"num_read_kb":2002944,"num_write":506299,"num_write_kb":2025196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4159,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:31.671530+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:25:31.671530+0000","last_peered":"2023-10-21T05:25:31.671530+0000","last_clean":"2023-10-21T05:25:31.671530+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:25:31.671530+0000","last_undegraded":"2023-10-21T05:25:31.671530+0000","last_fullsized":"2023-10-21T05:25:31.671530+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'531247","reported_seq":1058357,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:25:32.744594+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:25:32.744594+0000","last_peered":"2023-10-21T05:25:32.744594+0000","last_clean":"2023-10-21T05:25:32.744594+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:25:32.744594+0000","last_undegraded":"2023-10-21T05:25:32.744594+0000","last_fullsized":"2023-10-21T05:25:32.744594+0000","mapping_epoch":16,"log_start":"96'521226","ondisk_log_start":"96'521226","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2005,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 539s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6983815187,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":528135,"num_read_kb":2080996,"num_write":528304,"num_write_kb":2113210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18443","reported_seq":25326,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:32.230664+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:25:32.230664+0000","last_peered":"2023-10-21T05:25:32.230664+0000","last_clean":"2023-10-21T05:25:32.230664+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:25:32.230664+0000","last_undegraded":"2023-10-21T05:25:32.230664+0000","last_fullsized":"2023-10-21T05:25:32.230664+0000","mapping_epoch":13,"log_start":"80'8400","ondisk_log_start":"80'8400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":274,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.384655+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:22:01.384655+0000","last_peered":"2023-10-21T05:22:01.384655+0000","last_clean":"2023-10-21T05:22:01.384655+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:22:01.384655+0000","last_undegraded":"2023-10-21T05:22:01.384655+0000","last_fullsized":"2023-10-21T05:22:01.384655+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27872800787,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2115117,"num_read_kb":8335332,"num_write":2113008,"num_write_kb":8452026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17309995008,"data_stored":20203966502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40196,"ondisk_log_size":40196,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39237,"ondisk_log_size":39237,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739379,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10414644,"kb_used_data":8453036,"kb_used_omap":941,"kb_used_meta":1960658,"kb_avail":83957196,"statfs":{"total":96636764160,"available":85972168704,"internally_reserved":0,"allocated":8655908864,"data_stored":10102351817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":964399,"internal_metadata":2007714001},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:12 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.54400000000000004,"15min":0.60699999999999998},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.4249999999999998,"5min":11.939,"15min":11.939},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.23699999999999999,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":2.3980000000000001,"5min":7.6029999999999998,"15min":8.8510000000000009},"last":0.42599999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:25:12 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.56399999999999995,"15min":0.60999999999999999},"min":{"1min":0.215,"5min":0.20599999999999999,"15min":0.188},"max":{"1min":1.764,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40899999999999997,"15min":0.55300000000000005},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.19800000000000001},"max":{"1min":0.627,"5min":2.1299999999999999,"15min":8.4030000000000005},"last":0.41199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739377,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5792420,"kb_used_data":4279964,"kb_used_omap":478,"kb_used_meta":1511969,"kb_avail":88579420,"statfs":{"total":96636764160,"available":90705326080,"internally_reserved":0,"allocated":4382683136,"data_stored":5128556571,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1548256370},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,15,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.59699999999999998,"15min":0.51800000000000002},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.186},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.60099999999999998,"15min":0.51600000000000001},"min":{"1min":0.25,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":1.46,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.25}]},{"osd":2,"last update":"Sat Oct 21 05:25:16 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40000000000000002,"15min":0.42199999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":1.2969999999999999,"5min":2.4889999999999999,"15min":8.7669999999999995},"last":0.221},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.45700000000000002,"15min":0.41199999999999998},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.221},"max":{"1min":1.405,"5min":2.5870000000000002,"15min":4.1920000000000002},"last":0.28599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5925528,"kb_used_data":4173520,"kb_used_omap":504,"kb_used_meta":1751495,"kb_avail":88446312,"statfs":{"total":96636764160,"available":90569023488,"internally_reserved":0,"allocated":4273684480,"data_stored":4974545900,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1793531257},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,10,7,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.58499999999999996,"15min":0.51400000000000001},"min":{"1min":0.19400000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.67700000000000005,"5min":6.0030000000000001,"15min":15.727},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.754,"15min":0.60699999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.9630000000000001,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.254}]},{"osd":2,"last update":"Sat Oct 21 05:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.746,"15min":0.60399999999999998},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":11.337999999999999,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.754,"15min":0.66600000000000004},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.24199999999999999},"max":{"1min":0.56899999999999995,"5min":9.8680000000000003,"15min":10.967000000000001},"last":0.501}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4273233920,"data_stored":4974170112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4381650944,"data_stored":5127700499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8655110144,"data_stored":10102095891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:25:34.912 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:25:34.912 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:25:54.913 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:25:55.560 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:25:55.561 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:25:55.590 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2627,"stamp":"2023-10-21T05:25:54.307106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27898109057,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2141782,"num_read_kb":8381721,"num_write":2120868,"num_write_kb":8482715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79534,"ondisk_log_size":79534,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21121000,"kb_used_data":16963144,"kb_used_omap":1985,"kb_used_meta":4155838,"kb_avail":261994520,"statfs":{"total":289910292480,"available":268282388480,"internally_reserved":0,"allocated":17370259456,"data_stored":20263437264,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2033642,"internal_metadata":4255578134},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,7,9,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15327232,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4711,"num_read_kb":18676,"num_write":4401,"num_write_kb":17604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006174"},"pg_stats":[{"pgid":"2.7","version":"96'14821","reported_seq":20270,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:52.072685+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:25:52.072685+0000","last_peered":"2023-10-21T05:25:52.072685+0000","last_clean":"2023-10-21T05:25:52.072685+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:25:52.072685+0000","last_undegraded":"2023-10-21T05:25:52.072685+0000","last_fullsized":"2023-10-21T05:25:52.072685+0000","mapping_epoch":13,"log_start":"80'4800","ondisk_log_start":"80'4800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7556","reported_seq":11325,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:52.072193+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:25:52.072193+0000","last_peered":"2023-10-21T05:25:52.072193+0000","last_clean":"2023-10-21T05:25:52.072193+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:25:52.072193+0000","last_undegraded":"2023-10-21T05:25:52.072193+0000","last_fullsized":"2023-10-21T05:25:52.072193+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7556,"log_dups_size":0,"ondisk_log_size":7556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3787","reported_seq":5317,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:50.507772+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:25:50.507772+0000","last_peered":"2023-10-21T05:25:50.507772+0000","last_clean":"2023-10-21T05:25:50.507772+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:25:50.507772+0000","last_undegraded":"2023-10-21T05:25:50.507772+0000","last_fullsized":"2023-10-21T05:25:50.507772+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3787,"log_dups_size":0,"ondisk_log_size":3787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7502","reported_seq":10681,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:52.072525+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:25:52.072525+0000","last_peered":"2023-10-21T05:25:52.072525+0000","last_clean":"2023-10-21T05:25:52.072525+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:25:52.072525+0000","last_undegraded":"2023-10-21T05:25:52.072525+0000","last_fullsized":"2023-10-21T05:25:52.072525+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7502,"log_dups_size":0,"ondisk_log_size":7502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'542223","reported_seq":1078558,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:51.133896+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:25:51.133896+0000","last_peered":"2023-10-21T05:25:51.133896+0000","last_clean":"2023-10-21T05:25:51.133896+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:25:51.133896+0000","last_undegraded":"2023-10-21T05:25:51.133896+0000","last_fullsized":"2023-10-21T05:25:51.133896+0000","mapping_epoch":16,"log_start":"96'532218","ondisk_log_start":"96'532218","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7065735168,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":537386,"num_read_kb":2118080,"num_write":538732,"num_write_kb":2154928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":229,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:19.994954+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:23:19.994954+0000","last_peered":"2023-10-21T05:23:19.994954+0000","last_clean":"2023-10-21T05:23:19.994954+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:23:19.994954+0000","last_undegraded":"2023-10-21T05:23:19.994954+0000","last_fullsized":"2023-10-21T05:23:19.994954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'546890","reported_seq":1090980,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:52.871781+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:25:52.871781+0000","last_peered":"2023-10-21T05:25:52.871781+0000","last_clean":"2023-10-21T05:25:52.871781+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:25:52.871781+0000","last_undegraded":"2023-10-21T05:25:52.871781+0000","last_fullsized":"2023-10-21T05:25:52.871781+0000","mapping_epoch":16,"log_start":"96'536819","ondisk_log_start":"96'536819","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7189045248,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":545209,"num_read_kb":2148672,"num_write":543254,"num_write_kb":2173016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4236,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:52.092213+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:25:52.092213+0000","last_peered":"2023-10-21T05:25:52.092213+0000","last_clean":"2023-10-21T05:25:52.092213+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:25:52.092213+0000","last_undegraded":"2023-10-21T05:25:52.092213+0000","last_fullsized":"2023-10-21T05:25:52.092213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'511322","reported_seq":1020688,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:25:51.113833+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:25:51.113833+0000","last_peered":"2023-10-21T05:25:51.113833+0000","last_clean":"2023-10-21T05:25:51.113833+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:25:51.113833+0000","last_undegraded":"2023-10-21T05:25:51.113833+0000","last_fullsized":"2023-10-21T05:25:51.113833+0000","mapping_epoch":16,"log_start":"96'501320","ondisk_log_start":"96'501320","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":1338,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 308s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6652493824,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":510114,"num_read_kb":2010304,"num_write":508017,"num_write_kb":2032068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4171,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:52.074974+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:25:52.074974+0000","last_peered":"2023-10-21T05:25:52.074974+0000","last_clean":"2023-10-21T05:25:52.074974+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:25:52.074974+0000","last_undegraded":"2023-10-21T05:25:52.074974+0000","last_fullsized":"2023-10-21T05:25:52.074974+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'533107","reported_seq":1062079,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:25:52.877044+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:25:52.877044+0000","last_peered":"2023-10-21T05:25:52.877044+0000","last_clean":"2023-10-21T05:25:52.877044+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:25:52.877044+0000","last_undegraded":"2023-10-21T05:25:52.877044+0000","last_fullsized":"2023-10-21T05:25:52.877044+0000","mapping_epoch":16,"log_start":"96'523027","ondisk_log_start":"96'523027","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2066,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 559s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6990241811,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":529985,"num_read_kb":2088344,"num_write":530127,"num_write_kb":2120502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18493","reported_seq":25396,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:52.073604+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:25:52.073604+0000","last_peered":"2023-10-21T05:25:52.073604+0000","last_clean":"2023-10-21T05:25:52.073604+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:25:52.073604+0000","last_undegraded":"2023-10-21T05:25:52.073604+0000","last_fullsized":"2023-10-21T05:25:52.073604+0000","mapping_epoch":13,"log_start":"80'8400","ondisk_log_start":"80'8400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":274,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:22:01.384655+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:22:01.384655+0000","last_peered":"2023-10-21T05:22:01.384655+0000","last_clean":"2023-10-21T05:22:01.384655+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:22:01.384655+0000","last_undegraded":"2023-10-21T05:22:01.384655+0000","last_fullsized":"2023-10-21T05:22:01.384655+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27897516051,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2122694,"num_read_kb":8365400,"num_write":2120130,"num_write_kb":8480514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17368186880,"data_stored":20262158374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40158,"ondisk_log_size":40158,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39277,"ondisk_log_size":39277,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739383,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9204384,"kb_used_data":8481104,"kb_used_omap":1002,"kb_used_meta":722261,"kb_avail":85167456,"statfs":{"total":96636764160,"available":87211474944,"internally_reserved":0,"allocated":8684650496,"data_stored":10131093449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":739595819},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:12 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.54400000000000004,"15min":0.60699999999999998},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.4249999999999998,"5min":11.939,"15min":11.939},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.23699999999999999,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":2.3980000000000001,"5min":7.6029999999999998,"15min":8.8510000000000009},"last":0.26700000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:25:12 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.56399999999999995,"15min":0.60999999999999999},"min":{"1min":0.215,"5min":0.20599999999999999,"15min":0.188},"max":{"1min":1.764,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.501},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40899999999999997,"15min":0.55300000000000005},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.19800000000000001},"max":{"1min":0.627,"5min":2.1299999999999999,"15min":8.4030000000000005},"last":0.434}]}]},{"osd":1,"up_from":8,"seq":34359739381,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5850372,"kb_used_data":4294460,"kb_used_omap":478,"kb_used_meta":1555425,"kb_avail":88521468,"statfs":{"total":96636764160,"available":90645983232,"internally_reserved":0,"allocated":4397527040,"data_stored":5143400475,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1592755314},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.59699999999999998,"15min":0.51800000000000002},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.60099999999999998,"15min":0.51600000000000001},"min":{"1min":0.25,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":1.46,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Oct 21 05:25:16 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40000000000000002,"15min":0.42199999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":1.2969999999999999,"5min":2.4889999999999999,"15min":8.7669999999999995},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.45700000000000002,"15min":0.41199999999999998},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.221},"max":{"1min":1.405,"5min":2.5870000000000002,"15min":4.1920000000000002},"last":0.434}]}]},{"osd":0,"up_from":8,"seq":34359739384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6066244,"kb_used_data":4187580,"kb_used_omap":504,"kb_used_meta":1878151,"kb_avail":88305596,"statfs":{"total":96636764160,"available":90424930304,"internally_reserved":0,"allocated":4288081920,"data_stored":4988943340,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1923227001},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,5,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.58499999999999996,"15min":0.51400000000000001},"min":{"1min":0.19400000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.67700000000000005,"5min":6.0030000000000001,"15min":15.727},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.754,"15min":0.60699999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.9630000000000001,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.76900000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.746,"15min":0.60399999999999998},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":11.337999999999999,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.754,"15min":0.66600000000000004},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.24199999999999999},"max":{"1min":0.56899999999999995,"5min":9.8680000000000003,"15min":10.967000000000001},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4287488000,"data_stored":4988424192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4396564480,"data_stored":5142614035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8684134400,"data_stored":10131120147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:25:55.591 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:25:55.592 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:26:15.593 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:26:16.221 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:26:16.222 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:26:16.277 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2637,"stamp":"2023-10-21T05:26:14.313845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27923348609,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2149623,"num_read_kb":8412889,"num_write":2128170,"num_write_kb":8511923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79651,"ondisk_log_size":79651,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21368820,"kb_used_data":17020760,"kb_used_omap":1985,"kb_used_meta":4346046,"kb_avail":261746700,"statfs":{"total":289910292480,"available":268028620800,"internally_reserved":0,"allocated":17429258240,"data_stored":20322436048,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2033642,"internal_metadata":4450351126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,6,7,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15728640,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4931,"num_read_kb":19588,"num_write":4558,"num_write_kb":18232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002167"},"pg_stats":[{"pgid":"2.7","version":"96'14865","reported_seq":20330,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:12.552543+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:26:12.552543+0000","last_peered":"2023-10-21T05:26:12.552543+0000","last_clean":"2023-10-21T05:26:12.552543+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:26:12.552543+0000","last_undegraded":"2023-10-21T05:26:12.552543+0000","last_fullsized":"2023-10-21T05:26:12.552543+0000","mapping_epoch":13,"log_start":"80'4800","ondisk_log_start":"80'4800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7578","reported_seq":11355,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:12.551963+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:26:12.551963+0000","last_peered":"2023-10-21T05:26:12.551963+0000","last_clean":"2023-10-21T05:26:12.551963+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:26:12.551963+0000","last_undegraded":"2023-10-21T05:26:12.551963+0000","last_fullsized":"2023-10-21T05:26:12.551963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7578,"log_dups_size":0,"ondisk_log_size":7578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3798","reported_seq":5332,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:09.267636+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:26:09.267636+0000","last_peered":"2023-10-21T05:26:09.267636+0000","last_clean":"2023-10-21T05:26:09.267636+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:26:09.267636+0000","last_undegraded":"2023-10-21T05:26:09.267636+0000","last_fullsized":"2023-10-21T05:26:09.267636+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3798,"log_dups_size":0,"ondisk_log_size":3798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7524","reported_seq":10711,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:12.551991+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:26:12.551991+0000","last_peered":"2023-10-21T05:26:12.551991+0000","last_clean":"2023-10-21T05:26:12.551991+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:26:12.551991+0000","last_undegraded":"2023-10-21T05:26:12.551991+0000","last_fullsized":"2023-10-21T05:26:12.551991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7524,"log_dups_size":0,"ondisk_log_size":7524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'544122","reported_seq":1082480,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:11.277156+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:26:11.277156+0000","last_peered":"2023-10-21T05:26:11.277156+0000","last_clean":"2023-10-21T05:26:11.277156+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:26:11.277156+0000","last_undegraded":"2023-10-21T05:26:11.277156+0000","last_fullsized":"2023-10-21T05:26:11.277156+0000","mapping_epoch":16,"log_start":"96'534118","ondisk_log_start":"96'534118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7072194560,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":539405,"num_read_kb":2126120,"num_write":540631,"num_write_kb":2162524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":229,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:19.994954+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:23:19.994954+0000","last_peered":"2023-10-21T05:23:19.994954+0000","last_clean":"2023-10-21T05:23:19.994954+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:23:19.994954+0000","last_undegraded":"2023-10-21T05:23:19.994954+0000","last_fullsized":"2023-10-21T05:23:19.994954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'548813","reported_seq":1094918,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:13.019411+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:26:13.019411+0000","last_peered":"2023-10-21T05:26:13.019411+0000","last_clean":"2023-10-21T05:26:13.019411+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:26:13.019411+0000","last_undegraded":"2023-10-21T05:26:13.019411+0000","last_fullsized":"2023-10-21T05:26:13.019411+0000","mapping_epoch":16,"log_start":"96'538719","ondisk_log_start":"96'538719","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7195762688,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":547222,"num_read_kb":2156640,"num_write":545177,"num_write_kb":2180708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4245,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:07.436090+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:26:07.436090+0000","last_peered":"2023-10-21T05:26:07.436090+0000","last_clean":"2023-10-21T05:26:07.436090+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:26:07.436090+0000","last_undegraded":"2023-10-21T05:26:07.436090+0000","last_fullsized":"2023-10-21T05:26:07.436090+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'512965","reported_seq":1024208,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:26:11.279768+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:26:11.279768+0000","last_peered":"2023-10-21T05:26:11.279768+0000","last_clean":"2023-10-21T05:26:11.279768+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:26:11.279768+0000","last_undegraded":"2023-10-21T05:26:11.279768+0000","last_fullsized":"2023-10-21T05:26:11.279768+0000","mapping_epoch":16,"log_start":"96'502920","ondisk_log_start":"96'502920","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":1446,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 328s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6658150400,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":511976,"num_read_kb":2017720,"num_write":509655,"num_write_kb":2038620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4181,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:07.433274+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:26:07.433274+0000","last_peered":"2023-10-21T05:26:07.433274+0000","last_clean":"2023-10-21T05:26:07.433274+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:26:07.433274+0000","last_undegraded":"2023-10-21T05:26:07.433274+0000","last_fullsized":"2023-10-21T05:26:07.433274+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'535005","reported_seq":1065934,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:26:13.011644+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:26:13.011644+0000","last_peered":"2023-10-21T05:26:13.011644+0000","last_clean":"2023-10-21T05:26:13.011644+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:26:13.011644+0000","last_undegraded":"2023-10-21T05:26:13.011644+0000","last_fullsized":"2023-10-21T05:26:13.011644+0000","mapping_epoch":16,"log_start":"96'524927","ondisk_log_start":"96'524927","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2154,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 579s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":6996647955,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":531932,"num_read_kb":2096088,"num_write":531969,"num_write_kb":2127870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18548","reported_seq":25471,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:12.552666+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:26:12.552666+0000","last_peered":"2023-10-21T05:26:12.552666+0000","last_clean":"2023-10-21T05:26:12.552666+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:26:12.552666+0000","last_undegraded":"2023-10-21T05:26:12.552666+0000","last_fullsized":"2023-10-21T05:26:12.552666+0000","mapping_epoch":13,"log_start":"80'8500","ondisk_log_start":"80'8500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":275,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:02.133840+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:24:02.133840+0000","last_peered":"2023-10-21T05:24:02.133840+0000","last_clean":"2023-10-21T05:24:02.133840+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:24:02.133840+0000","last_undegraded":"2023-10-21T05:24:02.133840+0000","last_fullsized":"2023-10-21T05:24:02.133840+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27922755603,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2130535,"num_read_kb":8396568,"num_write":2127432,"num_write_kb":8509722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17427955712,"data_stored":20321927206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40221,"ondisk_log_size":40221,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39331,"ondisk_log_size":39331,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739387,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9315080,"kb_used_data":8509888,"kb_used_omap":1002,"kb_used_meta":804181,"kb_avail":85056760,"statfs":{"total":96636764160,"available":87098122240,"internally_reserved":0,"allocated":8714125312,"data_stored":10160568265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":823481899},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:12 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.54400000000000004,"15min":0.60699999999999998},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.4249999999999998,"5min":11.939,"15min":11.939},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.23699999999999999,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":2.3980000000000001,"5min":7.6029999999999998,"15min":8.8510000000000009},"last":0.309}]},{"osd":1,"last update":"Sat Oct 21 05:25:12 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.56399999999999995,"15min":0.60999999999999999},"min":{"1min":0.215,"5min":0.20599999999999999,"15min":0.188},"max":{"1min":1.764,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40899999999999997,"15min":0.55300000000000005},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.19800000000000001},"max":{"1min":0.627,"5min":2.1299999999999999,"15min":8.4030000000000005},"last":0.371}]}]},{"osd":1,"up_from":8,"seq":34359739385,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5919320,"kb_used_data":4309264,"kb_used_omap":478,"kb_used_meta":1609569,"kb_avail":88452520,"statfs":{"total":96636764160,"available":90575380480,"internally_reserved":0,"allocated":4412686336,"data_stored":5158559771,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1648198770},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:25:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.59699999999999998,"15min":0.51800000000000002},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":1.2490000000000001,"5min":6.8129999999999997,"15min":6.8129999999999997},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.60099999999999998,"15min":0.51600000000000001},"min":{"1min":0.25,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":1.46,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.64400000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:25:16 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40000000000000002,"15min":0.42199999999999999},"min":{"1min":0.23699999999999999,"5min":0.21299999999999999,"15min":0.20399999999999999},"max":{"1min":1.2969999999999999,"5min":2.4889999999999999,"15min":8.7669999999999995},"last":0.372},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.45700000000000002,"15min":0.41199999999999998},"min":{"1min":0.28299999999999997,"5min":0.223,"15min":0.221},"max":{"1min":1.405,"5min":2.5870000000000002,"15min":4.1920000000000002},"last":0.33700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6134420,"kb_used_data":4201608,"kb_used_omap":504,"kb_used_meta":1932295,"kb_avail":88237420,"statfs":{"total":96636764160,"available":90355118080,"internally_reserved":0,"allocated":4302446592,"data_stored":5003308012,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1978670457},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.58499999999999996,"15min":0.51400000000000001},"min":{"1min":0.19400000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.67700000000000005,"5min":6.0030000000000001,"15min":15.727},"last":9.2560000000000002},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.754,"15min":0.60699999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.9630000000000001,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.215}]},{"osd":2,"last update":"Sat Oct 21 05:25:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.746,"15min":0.60399999999999998},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":11.337999999999999,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.754,"15min":0.66600000000000004},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.24199999999999999},"max":{"1min":0.56899999999999995,"5min":9.8680000000000003,"15min":10.967000000000001},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4301979648,"data_stored":5002915840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4411994112,"data_stored":5158043667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8713981952,"data_stored":10160967699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:26:16.279 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:26:16.280 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:26:36.281 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:26:36.786 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:26:36.787 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:26:36.822 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2648,"stamp":"2023-10-21T05:26:36.317571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27948539009,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2157197,"num_read_kb":8443053,"num_write":2135468,"num_write_kb":8541115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79625,"ondisk_log_size":79625,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21198796,"kb_used_data":17079084,"kb_used_omap":1985,"kb_used_meta":4117694,"kb_avail":261916724,"statfs":{"total":289910292480,"available":268202725376,"internally_reserved":0,"allocated":17488982016,"data_stored":20382159824,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2033642,"internal_metadata":4216518678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,7,18,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12328960,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3807,"num_read_kb":15164,"num_write":3615,"num_write_kb":14460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002032"},"pg_stats":[{"pgid":"2.7","version":"96'14929","reported_seq":20410,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:33.033039+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:26:33.033039+0000","last_peered":"2023-10-21T05:26:33.033039+0000","last_clean":"2023-10-21T05:26:33.033039+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:26:33.033039+0000","last_undegraded":"2023-10-21T05:26:33.033039+0000","last_fullsized":"2023-10-21T05:26:33.033039+0000","mapping_epoch":13,"log_start":"80'4900","ondisk_log_start":"80'4900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7610","reported_seq":11396,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:33.032752+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:26:33.032752+0000","last_peered":"2023-10-21T05:26:33.032752+0000","last_clean":"2023-10-21T05:26:33.032752+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:26:33.032752+0000","last_undegraded":"2023-10-21T05:26:33.032752+0000","last_fullsized":"2023-10-21T05:26:33.032752+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7610,"log_dups_size":0,"ondisk_log_size":7610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3814","reported_seq":5352,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:29.806945+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:26:29.806945+0000","last_peered":"2023-10-21T05:26:29.806945+0000","last_clean":"2023-10-21T05:26:29.806945+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:26:29.806945+0000","last_undegraded":"2023-10-21T05:26:29.806945+0000","last_fullsized":"2023-10-21T05:26:29.806945+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3814,"log_dups_size":0,"ondisk_log_size":3814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7556","reported_seq":10751,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:33.032295+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:26:33.032295+0000","last_peered":"2023-10-21T05:26:33.032295+0000","last_clean":"2023-10-21T05:26:33.032295+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:26:33.032295+0000","last_undegraded":"2023-10-21T05:26:33.032295+0000","last_fullsized":"2023-10-21T05:26:33.032295+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7556,"log_dups_size":0,"ondisk_log_size":7556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'545966","reported_seq":1086207,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:31.348636+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:26:31.348636+0000","last_peered":"2023-10-21T05:26:31.348636+0000","last_clean":"2023-10-21T05:26:31.348636+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:26:31.348636+0000","last_undegraded":"2023-10-21T05:26:31.348636+0000","last_fullsized":"2023-10-21T05:26:31.348636+0000","mapping_epoch":16,"log_start":"96'535918","ondisk_log_start":"96'535918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7078481920,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":541296,"num_read_kb":2133656,"num_write":542475,"num_write_kb":2169900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":229,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:19.994954+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:23:19.994954+0000","last_peered":"2023-10-21T05:23:19.994954+0000","last_clean":"2023-10-21T05:23:19.994954+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:23:19.994954+0000","last_undegraded":"2023-10-21T05:23:19.994954+0000","last_fullsized":"2023-10-21T05:23:19.994954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'550755","reported_seq":1098869,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:33.154191+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:26:33.154191+0000","last_peered":"2023-10-21T05:26:33.154191+0000","last_clean":"2023-10-21T05:26:33.154191+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:26:33.154191+0000","last_undegraded":"2023-10-21T05:26:33.154191+0000","last_fullsized":"2023-10-21T05:26:33.154191+0000","mapping_epoch":16,"log_start":"96'540719","ondisk_log_start":"96'540719","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7202615296,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":549230,"num_read_kb":2164624,"num_write":547119,"num_write_kb":2188476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4257,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:27.925728+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:26:27.925728+0000","last_peered":"2023-10-21T05:26:27.925728+0000","last_clean":"2023-10-21T05:26:27.925728+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:26:27.925728+0000","last_undegraded":"2023-10-21T05:26:27.925728+0000","last_fullsized":"2023-10-21T05:26:27.925728+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'514673","reported_seq":1027666,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:26:31.332809+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:26:31.332809+0000","last_peered":"2023-10-21T05:26:31.332809+0000","last_clean":"2023-10-21T05:26:31.332809+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:26:31.332809+0000","last_undegraded":"2023-10-21T05:26:31.332809+0000","last_fullsized":"2023-10-21T05:26:31.332809+0000","mapping_epoch":16,"log_start":"96'504620","ondisk_log_start":"96'504620","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":1554,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 348s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6663852032,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":513713,"num_read_kb":2024640,"num_write":511355,"num_write_kb":2045420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4193,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:27.913139+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:26:27.913139+0000","last_peered":"2023-10-21T05:26:27.913139+0000","last_clean":"2023-10-21T05:26:27.913139+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:26:27.913139+0000","last_undegraded":"2023-10-21T05:26:27.913139+0000","last_fullsized":"2023-10-21T05:26:27.913139+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'536861","reported_seq":1069740,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:26:33.155028+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:26:33.155028+0000","last_peered":"2023-10-21T05:26:33.155028+0000","last_clean":"2023-10-21T05:26:33.155028+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:26:33.155028+0000","last_undegraded":"2023-10-21T05:26:33.155028+0000","last_fullsized":"2023-10-21T05:26:33.155028+0000","mapping_epoch":16,"log_start":"96'526827","ondisk_log_start":"96'526827","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2220,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 599s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":7002996755,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":533870,"num_read_kb":2103812,"num_write":533781,"num_write_kb":2135118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18628","reported_seq":25571,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:33.035285+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:26:33.035285+0000","last_peered":"2023-10-21T05:26:33.035285+0000","last_clean":"2023-10-21T05:26:33.035285+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:26:33.035285+0000","last_undegraded":"2023-10-21T05:26:33.035285+0000","last_fullsized":"2023-10-21T05:26:33.035285+0000","mapping_epoch":13,"log_start":"80'8600","ondisk_log_start":"80'8600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":275,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:02.133840+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:24:02.133840+0000","last_peered":"2023-10-21T05:24:02.133840+0000","last_clean":"2023-10-21T05:24:02.133840+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:24:02.133840+0000","last_undegraded":"2023-10-21T05:24:02.133840+0000","last_fullsized":"2023-10-21T05:24:02.133840+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27947946003,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2138109,"num_read_kb":8426732,"num_write":2134730,"num_write_kb":8538914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17487802368,"data_stored":20381773862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40171,"ondisk_log_size":40171,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39355,"ondisk_log_size":39355,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739392,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9169144,"kb_used_data":8538796,"kb_used_omap":1002,"kb_used_meta":629333,"kb_avail":85202696,"statfs":{"total":96636764160,"available":87247560704,"internally_reserved":0,"allocated":8743727104,"data_stored":10190170057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":644437547},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.55900000000000005,"15min":0.59899999999999998},"min":{"1min":0.255,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.9929999999999999,"5min":11.939,"15min":11.939},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.19900000000000001,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":0.57299999999999995,"5min":7.6029999999999998,"15min":8.8510000000000009},"last":0.20899999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.188},"max":{"1min":2.7480000000000002,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.43099999999999999,"15min":0.55700000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.19800000000000001},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":8.4030000000000005},"last":0.33100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739389,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5952552,"kb_used_data":4324444,"kb_used_omap":478,"kb_used_meta":1627617,"kb_avail":88419288,"statfs":{"total":96636764160,"available":90541350912,"internally_reserved":0,"allocated":4428230656,"data_stored":5174104091,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1666679922},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,9,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:20 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.59299999999999997,"15min":0.53400000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.183},"max":{"1min":6.4669999999999996,"5min":6.4669999999999996,"15min":6.8129999999999997},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.59799999999999998,"15min":0.53700000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":6.5890000000000004,"5min":6.5890000000000004,"15min":6.7110000000000003},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:26:20 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41099999999999998,"15min":0.42599999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":1.6759999999999999,"5min":2.4889999999999999,"15min":8.7669999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46600000000000003,"15min":0.41699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":1.718,"5min":2.5870000000000002,"15min":4.1920000000000002},"last":0.30199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739392,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6077100,"kb_used_data":4215844,"kb_used_omap":504,"kb_used_meta":1860743,"kb_avail":88294740,"statfs":{"total":96636764160,"available":90413813760,"internally_reserved":0,"allocated":4317024256,"data_stored":5017885676,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1905401209},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.66100000000000003,"15min":0.54600000000000004},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.2560000000000002,"5min":9.2560000000000002,"15min":15.727},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.80800000000000005,"15min":0.63100000000000001},"min":{"1min":0.215,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.5659999999999998,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.439}]},{"osd":2,"last update":"Sat Oct 21 05:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.79900000000000004,"15min":0.61399999999999999},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":5.5369999999999999,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.76400000000000001,"15min":0.65800000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.24199999999999999},"max":{"1min":5.6120000000000001,"5min":9.8680000000000003,"15min":10.967000000000001},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4316495872,"data_stored":5017432064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4427362304,"data_stored":5173411859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8743944192,"data_stored":10190929939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:26:36.824 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:26:36.824 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:26:51.161 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:26:51.154+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:26:51.169 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:26:51.162+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:26:51.169 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:26:51.162+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:26:51.170 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:26:51.162+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:26:56.826 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:26:57.439 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:26:57.440 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:26:57.499 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2658,"stamp":"2023-10-21T05:26:56.321436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27973565569,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2164747,"num_read_kb":8473125,"num_write":2142713,"num_write_kb":8570095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79771,"ondisk_log_size":79771,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21030532,"kb_used_data":17136748,"kb_used_omap":1985,"kb_used_meta":3891774,"kb_avail":262084988,"statfs":{"total":289910292480,"available":268375027712,"internally_reserved":0,"allocated":17548029952,"data_stored":20441207760,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2033642,"internal_metadata":3985176598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,7,9,16,20,19,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12378112,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3731,"num_read_kb":14864,"num_write":3571,"num_write_kb":14284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002476"},"pg_stats":[{"pgid":"2.7","version":"96'14974","reported_seq":20468,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:51.216223+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:26:51.216223+0000","last_peered":"2023-10-21T05:26:51.216223+0000","last_clean":"2023-10-21T05:26:51.216223+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:26:51.216223+0000","last_undegraded":"2023-10-21T05:26:51.216223+0000","last_fullsized":"2023-10-21T05:26:51.216223+0000","mapping_epoch":13,"log_start":"80'4900","ondisk_log_start":"80'4900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7633","reported_seq":11425,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:51.171406+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:26:51.171406+0000","last_peered":"2023-10-21T05:26:51.171406+0000","last_clean":"2023-10-21T05:26:51.171406+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:26:51.171406+0000","last_undegraded":"2023-10-21T05:26:51.171406+0000","last_fullsized":"2023-10-21T05:26:51.171406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7633,"log_dups_size":0,"ondisk_log_size":7633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3826","reported_seq":5367,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:51.159822+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:26:51.159822+0000","last_peered":"2023-10-21T05:26:51.159822+0000","last_clean":"2023-10-21T05:26:51.159822+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:26:51.159822+0000","last_undegraded":"2023-10-21T05:26:51.159822+0000","last_fullsized":"2023-10-21T05:26:51.159822+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3826,"log_dups_size":0,"ondisk_log_size":3826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7579","reported_seq":10780,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:51.166866+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:26:51.166866+0000","last_peered":"2023-10-21T05:26:51.166866+0000","last_clean":"2023-10-21T05:26:51.166866+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:26:51.166866+0000","last_undegraded":"2023-10-21T05:26:51.166866+0000","last_fullsized":"2023-10-21T05:26:51.166866+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7579,"log_dups_size":0,"ondisk_log_size":7579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'547805","reported_seq":1089997,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:51.442489+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:26:51.442489+0000","last_peered":"2023-10-21T05:26:51.442489+0000","last_clean":"2023-10-21T05:26:51.442489+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:26:51.442489+0000","last_undegraded":"2023-10-21T05:26:51.442489+0000","last_fullsized":"2023-10-21T05:26:51.442489+0000","mapping_epoch":16,"log_start":"96'537718","ondisk_log_start":"96'537718","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7084720128,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":543237,"num_read_kb":2141376,"num_write":544314,"num_write_kb":2177256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":229,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:19.994954+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:23:19.994954+0000","last_peered":"2023-10-21T05:23:19.994954+0000","last_clean":"2023-10-21T05:23:19.994954+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:23:19.994954+0000","last_undegraded":"2023-10-21T05:23:19.994954+0000","last_fullsized":"2023-10-21T05:23:19.994954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'552621","reported_seq":1102670,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:53.314664+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:26:53.314664+0000","last_peered":"2023-10-21T05:26:53.314664+0000","last_clean":"2023-10-21T05:26:53.314664+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:26:53.314664+0000","last_undegraded":"2023-10-21T05:26:53.314664+0000","last_fullsized":"2023-10-21T05:26:53.314664+0000","mapping_epoch":16,"log_start":"96'542619","ondisk_log_start":"96'542619","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7209168896,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":551167,"num_read_kb":2172340,"num_write":548985,"num_write_kb":2195940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4269,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:48.392679+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:26:48.392679+0000","last_peered":"2023-10-21T05:26:48.392679+0000","last_clean":"2023-10-21T05:26:48.392679+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:26:48.392679+0000","last_undegraded":"2023-10-21T05:26:48.392679+0000","last_fullsized":"2023-10-21T05:26:48.392679+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'516371","reported_seq":1031164,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:26:51.426399+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:26:51.426399+0000","last_peered":"2023-10-21T05:26:51.426399+0000","last_clean":"2023-10-21T05:26:51.426399+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:26:51.426399+0000","last_undegraded":"2023-10-21T05:26:51.426399+0000","last_fullsized":"2023-10-21T05:26:51.426399+0000","mapping_epoch":16,"log_start":"96'506320","ondisk_log_start":"96'506320","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":1649,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 368s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6669656064,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":515506,"num_read_kb":2031788,"num_write":513048,"num_write_kb":2052192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4205,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:48.392146+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:26:48.392146+0000","last_peered":"2023-10-21T05:26:48.392146+0000","last_clean":"2023-10-21T05:26:48.392146+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:26:48.392146+0000","last_undegraded":"2023-10-21T05:26:48.392146+0000","last_fullsized":"2023-10-21T05:26:48.392146+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'538746","reported_seq":1073514,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:26:53.334795+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:26:53.334795+0000","last_peered":"2023-10-21T05:26:53.334795+0000","last_clean":"2023-10-21T05:26:53.334795+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:26:53.334795+0000","last_undegraded":"2023-10-21T05:26:53.334795+0000","last_fullsized":"2023-10-21T05:26:53.334795+0000","mapping_epoch":16,"log_start":"96'528727","ondisk_log_start":"96'528727","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2278,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 619s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":7009427475,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":535749,"num_read_kb":2111300,"num_write":535628,"num_write_kb":2142506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18683","reported_seq":25641,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:51.246218+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:26:51.246218+0000","last_peered":"2023-10-21T05:26:51.246218+0000","last_clean":"2023-10-21T05:26:51.246218+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:26:51.246218+0000","last_undegraded":"2023-10-21T05:26:51.246218+0000","last_fullsized":"2023-10-21T05:26:51.246218+0000","mapping_epoch":13,"log_start":"80'8600","ondisk_log_start":"80'8600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":275,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:02.133840+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:24:02.133840+0000","last_peered":"2023-10-21T05:24:02.133840+0000","last_clean":"2023-10-21T05:24:02.133840+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:24:02.133840+0000","last_undegraded":"2023-10-21T05:24:02.133840+0000","last_fullsized":"2023-10-21T05:24:02.133840+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27972972563,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2145659,"num_read_kb":8456804,"num_write":2141975,"num_write_kb":8567894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17547243520,"data_stored":20441215014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40159,"ondisk_log_size":40159,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39513,"ondisk_log_size":39513,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739396,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9279856,"kb_used_data":8567592,"kb_used_omap":1002,"kb_used_meta":711253,"kb_avail":85091984,"statfs":{"total":96636764160,"available":87134191616,"internally_reserved":0,"allocated":8773214208,"data_stored":10219657161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":728323627},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.55900000000000005,"15min":0.59899999999999998},"min":{"1min":0.255,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.9929999999999999,"5min":11.939,"15min":11.939},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.19900000000000001,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":0.57299999999999995,"5min":7.6029999999999998,"15min":8.8510000000000009},"last":0.45600000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.188},"max":{"1min":2.7480000000000002,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.252},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.43099999999999999,"15min":0.55700000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.19800000000000001},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":8.4030000000000005},"last":0.58599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739393,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6042700,"kb_used_data":4339204,"kb_used_omap":478,"kb_used_meta":1703009,"kb_avail":88329140,"statfs":{"total":96636764160,"available":90449039360,"internally_reserved":0,"allocated":4443344896,"data_stored":5189218331,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1743881330},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,4,9,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:20 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.59299999999999997,"15min":0.53400000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.183},"max":{"1min":6.4669999999999996,"5min":6.4669999999999996,"15min":6.8129999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.59799999999999998,"15min":0.53700000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":6.5890000000000004,"5min":6.5890000000000004,"15min":6.7110000000000003},"last":0.42899999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:26:20 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41099999999999998,"15min":0.42599999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":1.6759999999999999,"5min":2.4889999999999999,"15min":8.7669999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46600000000000003,"15min":0.41699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":1.718,"5min":2.5870000000000002,"15min":4.1920000000000002},"last":0.38800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739396,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5707976,"kb_used_data":4229952,"kb_used_omap":504,"kb_used_meta":1477511,"kb_avail":88663864,"statfs":{"total":96636764160,"available":90791796736,"internally_reserved":0,"allocated":4331470848,"data_stored":5032332268,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1512971641},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,2,3,8,13,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.66100000000000003,"15min":0.54600000000000004},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.2560000000000002,"5min":9.2560000000000002,"15min":15.727},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.80800000000000005,"15min":0.63100000000000001},"min":{"1min":0.215,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.5659999999999998,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.375}]},{"osd":2,"last update":"Sat Oct 21 05:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.79900000000000004,"15min":0.61399999999999999},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":5.5369999999999999,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.76400000000000001,"15min":0.65800000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.24199999999999999},"max":{"1min":5.6120000000000001,"5min":9.8680000000000003,"15min":10.967000000000001},"last":0.33600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4330958848,"data_stored":5031895040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4442578944,"data_stored":5188628499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8773705728,"data_stored":10220691475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:26:57.501 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:26:57.502 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:27:17.503 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:27:17.999 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:27:18.000 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:27:18.028 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2668,"stamp":"2023-10-21T05:27:16.324928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27999181953,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2172472,"num_read_kb":8503925,"num_write":2150135,"num_write_kb":8599783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79762,"ondisk_log_size":79762,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21321424,"kb_used_data":17197232,"kb_used_omap":1985,"kb_used_meta":4122174,"kb_avail":261794096,"statfs":{"total":289910292480,"available":268077154304,"internally_reserved":0,"allocated":17609965568,"data_stored":20503143376,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2033642,"internal_metadata":4221106198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,3,13,9,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12886016,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3763,"num_read_kb":15016,"num_write":3761,"num_write_kb":15044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002100"},"pg_stats":[{"pgid":"2.7","version":"96'15014","reported_seq":20524,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:12.265630+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:27:12.265630+0000","last_peered":"2023-10-21T05:27:12.265630+0000","last_clean":"2023-10-21T05:27:12.265630+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:27:12.265630+0000","last_undegraded":"2023-10-21T05:27:12.265630+0000","last_fullsized":"2023-10-21T05:27:12.265630+0000","mapping_epoch":13,"log_start":"80'5000","ondisk_log_start":"80'5000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7653","reported_seq":11453,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:12.253878+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:27:12.253878+0000","last_peered":"2023-10-21T05:27:12.253878+0000","last_clean":"2023-10-21T05:27:12.253878+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:27:12.253878+0000","last_undegraded":"2023-10-21T05:27:12.253878+0000","last_fullsized":"2023-10-21T05:27:12.253878+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7653,"log_dups_size":0,"ondisk_log_size":7653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3836","reported_seq":5381,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:10.724456+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:27:10.724456+0000","last_peered":"2023-10-21T05:27:10.724456+0000","last_clean":"2023-10-21T05:27:10.724456+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:27:10.724456+0000","last_undegraded":"2023-10-21T05:27:10.724456+0000","last_fullsized":"2023-10-21T05:27:10.724456+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3836,"log_dups_size":0,"ondisk_log_size":3836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7599","reported_seq":10808,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:12.254010+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:27:12.254010+0000","last_peered":"2023-10-21T05:27:12.254010+0000","last_clean":"2023-10-21T05:27:12.254010+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:27:12.254010+0000","last_undegraded":"2023-10-21T05:27:12.254010+0000","last_fullsized":"2023-10-21T05:27:12.254010+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7599,"log_dups_size":0,"ondisk_log_size":7599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'549712","reported_seq":1093869,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:11.523720+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:27:11.523720+0000","last_peered":"2023-10-21T05:27:11.523720+0000","last_clean":"2023-10-21T05:27:11.523720+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:27:11.523720+0000","last_undegraded":"2023-10-21T05:27:11.523720+0000","last_fullsized":"2023-10-21T05:27:11.523720+0000","mapping_epoch":16,"log_start":"96'539618","ondisk_log_start":"96'539618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7091326976,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":545202,"num_read_kb":2149204,"num_write":546221,"num_write_kb":2184884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":229,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:23:19.994954+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:23:19.994954+0000","last_peered":"2023-10-21T05:23:19.994954+0000","last_clean":"2023-10-21T05:23:19.994954+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:23:19.994954+0000","last_undegraded":"2023-10-21T05:23:19.994954+0000","last_fullsized":"2023-10-21T05:23:19.994954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'554489","reported_seq":1106611,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:13.517708+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:27:13.517708+0000","last_peered":"2023-10-21T05:27:13.517708+0000","last_clean":"2023-10-21T05:27:13.517708+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:27:13.517708+0000","last_undegraded":"2023-10-21T05:27:13.517708+0000","last_fullsized":"2023-10-21T05:27:13.517708+0000","mapping_epoch":16,"log_start":"96'544419","ondisk_log_start":"96'544419","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7215710208,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":553237,"num_read_kb":2180600,"num_write":550853,"num_write_kb":2203412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4281,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:08.878386+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:27:08.878386+0000","last_peered":"2023-10-21T05:27:08.878386+0000","last_clean":"2023-10-21T05:27:08.878386+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:27:08.878386+0000","last_undegraded":"2023-10-21T05:27:08.878386+0000","last_fullsized":"2023-10-21T05:27:08.878386+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'518159","reported_seq":1034751,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:27:11.524857+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:27:11.524857+0000","last_peered":"2023-10-21T05:27:11.524857+0000","last_clean":"2023-10-21T05:27:11.524857+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:27:11.524857+0000","last_undegraded":"2023-10-21T05:27:11.524857+0000","last_fullsized":"2023-10-21T05:27:11.524857+0000","mapping_epoch":16,"log_start":"96'508120","ondisk_log_start":"96'508120","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":1739,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 388s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6675595264,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":517291,"num_read_kb":2038900,"num_write":514833,"num_write_kb":2059332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4217,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:08.872468+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:27:08.872468+0000","last_peered":"2023-10-21T05:27:08.872468+0000","last_clean":"2023-10-21T05:27:08.872468+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:27:08.872468+0000","last_undegraded":"2023-10-21T05:27:08.872468+0000","last_fullsized":"2023-10-21T05:27:08.872468+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'540634","reported_seq":1077315,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:27:13.526692+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:27:13.526692+0000","last_peered":"2023-10-21T05:27:13.526692+0000","last_clean":"2023-10-21T05:27:13.526692+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:27:13.526692+0000","last_undegraded":"2023-10-21T05:27:13.526692+0000","last_fullsized":"2023-10-21T05:27:13.526692+0000","mapping_epoch":16,"log_start":"96'530627","ondisk_log_start":"96'530627","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2329,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 640s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":7015956499,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":537654,"num_read_kb":2118900,"num_write":537490,"num_write_kb":2149954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18733","reported_seq":25711,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:12.315900+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:27:12.315900+0000","last_peered":"2023-10-21T05:27:12.315900+0000","last_clean":"2023-10-21T05:27:12.315900+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:27:12.315900+0000","last_undegraded":"2023-10-21T05:27:12.315900+0000","last_fullsized":"2023-10-21T05:27:12.315900+0000","mapping_epoch":13,"log_start":"80'8700","ondisk_log_start":"80'8700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":275,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:02.133840+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:24:02.133840+0000","last_peered":"2023-10-21T05:24:02.133840+0000","last_clean":"2023-10-21T05:24:02.133840+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:24:02.133840+0000","last_undegraded":"2023-10-21T05:24:02.133840+0000","last_fullsized":"2023-10-21T05:24:02.133840+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27998588947,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2153384,"num_read_kb":8487604,"num_write":2149397,"num_write_kb":8597582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17607921664,"data_stored":20501893158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40210,"ondisk_log_size":40210,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39453,"ondisk_log_size":39453,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739400,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9432936,"kb_used_data":8598496,"kb_used_omap":1002,"kb_used_meta":833429,"kb_avail":84938904,"statfs":{"total":96636764160,"available":86977437696,"internally_reserved":0,"allocated":8804859904,"data_stored":10251302857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":853431851},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.55900000000000005,"15min":0.59899999999999998},"min":{"1min":0.255,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.9929999999999999,"5min":11.939,"15min":11.939},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.19900000000000001,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":0.57299999999999995,"5min":7.6029999999999998,"15min":8.8510000000000009},"last":0.26700000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.188},"max":{"1min":2.7480000000000002,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.43099999999999999,"15min":0.55700000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.19800000000000001},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":8.4030000000000005},"last":0.42899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739397,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6111644,"kb_used_data":4354064,"kb_used_omap":478,"kb_used_meta":1757089,"kb_avail":88260196,"statfs":{"total":96636764160,"available":90378440704,"internally_reserved":0,"allocated":4458561536,"data_stored":5204434971,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1799259250},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,7,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:26:20 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.59299999999999997,"15min":0.53400000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.183},"max":{"1min":6.4669999999999996,"5min":6.4669999999999996,"15min":6.8129999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.59799999999999998,"15min":0.53700000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":6.5890000000000004,"5min":6.5890000000000004,"15min":6.7110000000000003},"last":0.60899999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:26:20 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41099999999999998,"15min":0.42599999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":1.6759999999999999,"5min":2.4889999999999999,"15min":8.7669999999999995},"last":0.37},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46600000000000003,"15min":0.41699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":1.718,"5min":2.5870000000000002,"15min":4.1920000000000002},"last":0.33100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739400,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5776844,"kb_used_data":4244672,"kb_used_omap":504,"kb_used_meta":1531655,"kb_avail":88594996,"statfs":{"total":96636764160,"available":90721275904,"internally_reserved":0,"allocated":4346544128,"data_stored":5047405548,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1568415097},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.66100000000000003,"15min":0.54600000000000004},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.2560000000000002,"5min":9.2560000000000002,"15min":15.727},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.80800000000000005,"15min":0.63100000000000001},"min":{"1min":0.215,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":5.5659999999999998,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.308}]},{"osd":2,"last update":"Sat Oct 21 05:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.79900000000000004,"15min":0.61399999999999999},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":5.5369999999999999,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.76400000000000001,"15min":0.65800000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.24199999999999999},"max":{"1min":5.6120000000000001,"5min":9.8680000000000003,"15min":10.967000000000001},"last":0.42399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4346077184,"data_stored":5047013376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4457852928,"data_stored":5203902483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8803991552,"data_stored":10250977299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:27:18.031 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:27:18.032 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:27:18.643 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:27:18.684 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:27:20.227 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:20.218+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:27:20.236 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:20.230+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:27:20.237 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:20.230+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882f70 2023-10-21T05:27:20.238 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:20.230+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b883110 2023-10-21T05:27:20.238 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:20.230+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:27:20.244 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:20.238+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfca90 2023-10-21T05:27:20.245 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:20.238+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:27:20.246 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:20.238+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bcabd40 2023-10-21T05:27:20.246 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:20.238+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bf43ee0 2023-10-21T05:27:20.247 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:20.238+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:27:20.247 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:20.238+0000 7fc58f19b640 -1 reset not still connected to 0x55f03c399110 2023-10-21T05:27:38.686 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:27:39.258 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:27:39.259 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:27:39.295 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2679,"stamp":"2023-10-21T05:27:38.330002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28027088001,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2181012,"num_read_kb":8537969,"num_write":2158208,"num_write_kb":8632075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79972,"ondisk_log_size":79972,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21520404,"kb_used_data":17265084,"kb_used_omap":1985,"kb_used_meta":4253310,"kb_avail":261595116,"statfs":{"total":289910292480,"available":267873398784,"internally_reserved":0,"allocated":17679446016,"data_stored":20572623824,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2033642,"internal_metadata":4355389462},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,9,11,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15548416,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4752,"num_read_kb":18952,"num_write":4493,"num_write_kb":17972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003335"},"pg_stats":[{"pgid":"2.7","version":"96'15062","reported_seq":20588,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:31.459484+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:27:31.459484+0000","last_peered":"2023-10-21T05:27:31.459484+0000","last_clean":"2023-10-21T05:27:31.459484+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:27:31.459484+0000","last_undegraded":"2023-10-21T05:27:31.459484+0000","last_fullsized":"2023-10-21T05:27:31.459484+0000","mapping_epoch":13,"log_start":"80'5000","ondisk_log_start":"80'5000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7677","reported_seq":11485,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:31.440120+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:27:31.440120+0000","last_peered":"2023-10-21T05:27:31.440120+0000","last_clean":"2023-10-21T05:27:31.440120+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:27:31.440120+0000","last_undegraded":"2023-10-21T05:27:31.440120+0000","last_fullsized":"2023-10-21T05:27:31.440120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7677,"log_dups_size":0,"ondisk_log_size":7677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3848","reported_seq":5398,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:31.432263+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:27:31.432263+0000","last_peered":"2023-10-21T05:27:31.432263+0000","last_clean":"2023-10-21T05:27:31.432263+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:27:31.432263+0000","last_undegraded":"2023-10-21T05:27:31.432263+0000","last_fullsized":"2023-10-21T05:27:31.432263+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7623","reported_seq":10841,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:31.436921+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:27:31.436921+0000","last_peered":"2023-10-21T05:27:31.436921+0000","last_clean":"2023-10-21T05:27:31.436921+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:27:31.436921+0000","last_undegraded":"2023-10-21T05:27:31.436921+0000","last_fullsized":"2023-10-21T05:27:31.436921+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7623,"log_dups_size":0,"ondisk_log_size":7623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'551976","reported_seq":1098556,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:36.641079+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:27:36.641079+0000","last_peered":"2023-10-21T05:27:36.641079+0000","last_clean":"2023-10-21T05:27:36.641079+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:27:36.641079+0000","last_undegraded":"2023-10-21T05:27:36.641079+0000","last_fullsized":"2023-10-21T05:27:36.641079+0000","mapping_epoch":16,"log_start":"96'541918","ondisk_log_start":"96'541918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7099154432,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":547626,"num_read_kb":2158848,"num_write":548485,"num_write_kb":2193940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":230,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:26.000205+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:25:26.000205+0000","last_peered":"2023-10-21T05:25:26.000205+0000","last_clean":"2023-10-21T05:25:26.000205+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:25:26.000205+0000","last_undegraded":"2023-10-21T05:25:26.000205+0000","last_fullsized":"2023-10-21T05:25:26.000205+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'556319","reported_seq":1110377,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:33.622910+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:27:33.622910+0000","last_peered":"2023-10-21T05:27:33.622910+0000","last_clean":"2023-10-21T05:27:33.622910+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:27:33.622910+0000","last_undegraded":"2023-10-21T05:27:33.622910+0000","last_fullsized":"2023-10-21T05:27:33.622910+0000","mapping_epoch":16,"log_start":"96'546219","ondisk_log_start":"96'546219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7222157312,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":555176,"num_read_kb":2188336,"num_write":552683,"num_write_kb":2210732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4297,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:34.482463+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:27:34.482463+0000","last_peered":"2023-10-21T05:27:34.482463+0000","last_clean":"2023-10-21T05:27:34.482463+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:27:34.482463+0000","last_undegraded":"2023-10-21T05:27:34.482463+0000","last_fullsized":"2023-10-21T05:27:34.482463+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'520332","reported_seq":1039213,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:27:36.630123+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:27:36.630123+0000","last_peered":"2023-10-21T05:27:36.630123+0000","last_clean":"2023-10-21T05:27:36.630123+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:27:36.630123+0000","last_undegraded":"2023-10-21T05:27:36.630123+0000","last_fullsized":"2023-10-21T05:27:36.630123+0000","mapping_epoch":16,"log_start":"96'510320","ondisk_log_start":"96'510320","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":1938,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 413s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6682877952,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":519561,"num_read_kb":2047956,"num_write":516998,"num_write_kb":2067992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4232,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:34.477276+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:27:34.477276+0000","last_peered":"2023-10-21T05:27:34.477276+0000","last_clean":"2023-10-21T05:27:34.477276+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:27:34.477276+0000","last_undegraded":"2023-10-21T05:27:34.477276+0000","last_fullsized":"2023-10-21T05:27:34.477276+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'542509","reported_seq":1081109,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:27:33.632382+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:27:33.632382+0000","last_peered":"2023-10-21T05:27:33.632382+0000","last_clean":"2023-10-21T05:27:33.632382+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:27:33.632382+0000","last_undegraded":"2023-10-21T05:27:33.632382+0000","last_fullsized":"2023-10-21T05:27:33.632382+0000","mapping_epoch":16,"log_start":"96'532427","ondisk_log_start":"96'532427","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2421,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 660s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":7022305299,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":539561,"num_read_kb":2126508,"num_write":539304,"num_write_kb":2157210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18793","reported_seq":25792,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:31.497009+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:27:31.497009+0000","last_peered":"2023-10-21T05:27:31.497009+0000","last_clean":"2023-10-21T05:27:31.497009+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:27:31.497009+0000","last_undegraded":"2023-10-21T05:27:31.497009+0000","last_fullsized":"2023-10-21T05:27:31.497009+0000","mapping_epoch":13,"log_start":"80'8700","ondisk_log_start":"80'8700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":275,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:02.133840+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:24:02.133840+0000","last_peered":"2023-10-21T05:24:02.133840+0000","last_clean":"2023-10-21T05:24:02.133840+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:24:02.133840+0000","last_undegraded":"2023-10-21T05:24:02.133840+0000","last_fullsized":"2023-10-21T05:24:02.133840+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28026494995,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2161924,"num_read_kb":8521648,"num_write":2157470,"num_write_kb":8629874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17677725696,"data_stored":20571697190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40252,"ondisk_log_size":40252,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39621,"ondisk_log_size":39621,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739405,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9509576,"kb_used_data":8634240,"kb_used_omap":1002,"kb_used_meta":874325,"kb_avail":84862264,"statfs":{"total":96636764160,"available":86898958336,"internally_reserved":0,"allocated":8841461760,"data_stored":10287904713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":895309355},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:27:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60899999999999999,"15min":0.59899999999999998},"min":{"1min":0.27800000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.2440000000000002,"5min":11.939,"15min":11.939},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.622,"15min":0.58099999999999996},"min":{"1min":0.20899999999999999,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":3.2570000000000001,"5min":7.6029999999999998,"15min":8.8510000000000009},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Oct 21 05:27:17 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.58799999999999997,"15min":0.56599999999999995},"min":{"1min":0.21199999999999999,"5min":0.20599999999999999,"15min":0.188},"max":{"1min":4.1550000000000002,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.48299999999999998,"15min":0.55700000000000005},"min":{"1min":0.23000000000000001,"5min":0.22700000000000001,"15min":0.19800000000000001},"max":{"1min":4.1680000000000001,"5min":4.1680000000000001,"15min":8.4030000000000005},"last":0.38800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739401,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6180232,"kb_used_data":4368512,"kb_used_omap":478,"kb_used_meta":1811233,"kb_avail":88191608,"statfs":{"total":96636764160,"available":90308206592,"internally_reserved":0,"allocated":4473356288,"data_stored":5219229723,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1854702706},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,5,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:27:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.59999999999999998,"15min":0.53300000000000003},"min":{"1min":0.191,"5min":0.186,"15min":0.183},"max":{"1min":2.855,"5min":6.4669999999999996,"15min":6.8129999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.624,"5min":0.63600000000000001,"15min":0.54500000000000004},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.298,"5min":6.5890000000000004,"15min":6.7110000000000003},"last":0.47299999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:27:21 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.41299999999999998,"15min":0.42699999999999999},"min":{"1min":0.22800000000000001,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":0.57899999999999996,"5min":2.4889999999999999,"15min":8.7669999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.49299999999999999,"15min":0.42499999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.3399999999999999,"5min":3.3399999999999999,"15min":4.1920000000000002},"last":0.50700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5830596,"kb_used_data":4262332,"kb_used_omap":504,"kb_used_meta":1567751,"kb_avail":88541244,"statfs":{"total":96636764160,"available":90666233856,"internally_reserved":0,"allocated":4364627968,"data_stored":5065489388,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1605377401},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.82299999999999995,"15min":0.59799999999999998},"min":{"1min":0.223,"5min":0.19400000000000001,"15min":0.17899999999999999},"max":{"1min":11.164,"5min":11.164,"15min":15.727},"last":1.6160000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.81999999999999995,"15min":0.63200000000000001},"min":{"1min":0.24199999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":2.282,"5min":6.0529999999999999,"15min":15.757999999999999},"last":1.712}]},{"osd":2,"last update":"Sat Oct 21 05:27:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.67500000000000004,"15min":0.59199999999999997},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22},"max":{"1min":0.47699999999999998,"5min":11.337999999999999,"15min":11.337999999999999},"last":1.6539999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.63400000000000001,"15min":0.63600000000000001},"min":{"1min":0.27500000000000002,"5min":0.25,"15min":0.24199999999999999},"max":{"1min":0.47899999999999998,"5min":5.6120000000000001,"15min":10.967000000000001},"last":1.681}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4364218368,"data_stored":5065154560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4472782848,"data_stored":5218832403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8840724480,"data_stored":10287710227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:27:39.297 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:27:39.298 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:27:51.228 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:51.218+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:27:51.229 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:51.218+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:27:51.229 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:27:51.218+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:27:59.299 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:27:59.775 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:27:59.777 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:27:59.808 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2689,"stamp":"2023-10-21T05:27:58.333822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28052507777,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2188532,"num_read_kb":8567977,"num_write":2165580,"num_write_kb":8661563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79851,"ondisk_log_size":79851,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21788956,"kb_used_data":17324100,"kb_used_omap":1985,"kb_used_meta":4462846,"kb_avail":261326564,"statfs":{"total":289910292480,"available":267598401536,"internally_reserved":0,"allocated":17739878400,"data_stored":20633056208,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2033642,"internal_metadata":4569954326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,7,11,10,13,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15851520,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4638,"num_read_kb":18500,"num_write":4614,"num_write_kb":18456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002384"},"pg_stats":[{"pgid":"2.7","version":"96'15107","reported_seq":20649,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:53.718241+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:27:53.718241+0000","last_peered":"2023-10-21T05:27:53.718241+0000","last_clean":"2023-10-21T05:27:53.718241+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:27:53.718241+0000","last_undegraded":"2023-10-21T05:27:53.718241+0000","last_fullsized":"2023-10-21T05:27:53.718241+0000","mapping_epoch":13,"log_start":"80'5100","ondisk_log_start":"80'5100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7701","reported_seq":11517,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:53.702477+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:27:53.702477+0000","last_peered":"2023-10-21T05:27:53.702477+0000","last_clean":"2023-10-21T05:27:53.702477+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:27:53.702477+0000","last_undegraded":"2023-10-21T05:27:53.702477+0000","last_fullsized":"2023-10-21T05:27:53.702477+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7701,"log_dups_size":0,"ondisk_log_size":7701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3860","reported_seq":5414,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:51.840277+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:27:51.840277+0000","last_peered":"2023-10-21T05:27:51.840277+0000","last_clean":"2023-10-21T05:27:51.840277+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:27:51.840277+0000","last_undegraded":"2023-10-21T05:27:51.840277+0000","last_fullsized":"2023-10-21T05:27:51.840277+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7646","reported_seq":10872,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:53.704099+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:27:53.704099+0000","last_peered":"2023-10-21T05:27:53.704099+0000","last_clean":"2023-10-21T05:27:53.704099+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:27:53.704099+0000","last_undegraded":"2023-10-21T05:27:53.704099+0000","last_fullsized":"2023-10-21T05:27:53.704099+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7646,"log_dups_size":0,"ondisk_log_size":7646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'553870","reported_seq":1102325,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:56.744569+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:27:56.744569+0000","last_peered":"2023-10-21T05:27:56.744569+0000","last_clean":"2023-10-21T05:27:56.744569+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:27:56.744569+0000","last_undegraded":"2023-10-21T05:27:56.744569+0000","last_fullsized":"2023-10-21T05:27:56.744569+0000","mapping_epoch":16,"log_start":"96'543818","ondisk_log_start":"96'543818","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7105576960,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":549501,"num_read_kb":2166336,"num_write":550379,"num_write_kb":2201516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":230,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:26.000205+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:25:26.000205+0000","last_peered":"2023-10-21T05:25:26.000205+0000","last_clean":"2023-10-21T05:25:26.000205+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:25:26.000205+0000","last_undegraded":"2023-10-21T05:25:26.000205+0000","last_fullsized":"2023-10-21T05:25:26.000205+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'558149","reported_seq":1114206,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:53.725838+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:27:53.725838+0000","last_peered":"2023-10-21T05:27:53.725838+0000","last_clean":"2023-10-21T05:27:53.725838+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:27:53.725838+0000","last_undegraded":"2023-10-21T05:27:53.725838+0000","last_fullsized":"2023-10-21T05:27:53.725838+0000","mapping_epoch":16,"log_start":"96'548119","ondisk_log_start":"96'548119","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7228608512,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":557172,"num_read_kb":2196316,"num_write":554513,"num_write_kb":2218052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4309,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:55.050934+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:27:55.050934+0000","last_peered":"2023-10-21T05:27:55.050934+0000","last_clean":"2023-10-21T05:27:55.050934+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:27:55.050934+0000","last_undegraded":"2023-10-21T05:27:55.050934+0000","last_fullsized":"2023-10-21T05:27:55.050934+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'522088","reported_seq":1042794,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:27:56.746475+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:27:56.746475+0000","last_peered":"2023-10-21T05:27:56.746475+0000","last_clean":"2023-10-21T05:27:56.746475+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:27:56.746475+0000","last_undegraded":"2023-10-21T05:27:56.746475+0000","last_fullsized":"2023-10-21T05:27:56.746475+0000","mapping_epoch":16,"log_start":"96'512020","ondisk_log_start":"96'512020","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2027,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 433s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6688882688,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":521376,"num_read_kb":2055188,"num_write":518752,"num_write_kb":2075008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4244,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:54.986042+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:27:54.986042+0000","last_peered":"2023-10-21T05:27:54.986042+0000","last_clean":"2023-10-21T05:27:54.986042+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:27:54.986042+0000","last_undegraded":"2023-10-21T05:27:54.986042+0000","last_fullsized":"2023-10-21T05:27:54.986042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'544448","reported_seq":1084895,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:27:53.712630+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:27:53.712630+0000","last_peered":"2023-10-21T05:27:53.712630+0000","last_clean":"2023-10-21T05:27:53.712630+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:27:53.712630+0000","last_undegraded":"2023-10-21T05:27:53.712630+0000","last_fullsized":"2023-10-21T05:27:53.712630+0000","mapping_epoch":16,"log_start":"96'534427","ondisk_log_start":"96'534427","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2502,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 680s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":7028846611,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":541395,"num_read_kb":2133816,"num_write":541198,"num_write_kb":2164786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18849","reported_seq":25868,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:53.719538+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:27:53.719538+0000","last_peered":"2023-10-21T05:27:53.719538+0000","last_clean":"2023-10-21T05:27:53.719538+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:27:53.719538+0000","last_undegraded":"2023-10-21T05:27:53.719538+0000","last_fullsized":"2023-10-21T05:27:53.719538+0000","mapping_epoch":13,"log_start":"80'8800","ondisk_log_start":"80'8800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":275,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:24:02.133840+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:24:02.133840+0000","last_peered":"2023-10-21T05:24:02.133840+0000","last_clean":"2023-10-21T05:24:02.133840+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:24:02.133840+0000","last_undegraded":"2023-10-21T05:24:02.133840+0000","last_fullsized":"2023-10-21T05:24:02.133840+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28051914771,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2169444,"num_read_kb":8551656,"num_write":2164842,"num_write_kb":8659362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17738211328,"data_stored":20632182822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40171,"ondisk_log_size":40171,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39581,"ondisk_log_size":39581,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739409,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9641728,"kb_used_data":8663672,"kb_used_omap":1002,"kb_used_meta":977045,"kb_avail":84730112,"statfs":{"total":96636764160,"available":86763634688,"internally_reserved":0,"allocated":8871600128,"data_stored":10318043081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":1000494635},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:27:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60899999999999999,"15min":0.59899999999999998},"min":{"1min":0.27800000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.2440000000000002,"5min":11.939,"15min":11.939},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.622,"15min":0.58099999999999996},"min":{"1min":0.20899999999999999,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":3.2570000000000001,"5min":7.6029999999999998,"15min":8.8510000000000009},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:27:17 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.58799999999999997,"15min":0.56599999999999995},"min":{"1min":0.21199999999999999,"5min":0.20599999999999999,"15min":0.188},"max":{"1min":4.1550000000000002,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.245},{"interface":"front","average":{"1min":0.625,"5min":0.48299999999999998,"15min":0.55700000000000005},"min":{"1min":0.23000000000000001,"5min":0.22700000000000001,"15min":0.19800000000000001},"max":{"1min":4.1680000000000001,"5min":4.1680000000000001,"15min":8.4030000000000005},"last":0.51100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739405,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6265872,"kb_used_data":4383432,"kb_used_omap":478,"kb_used_meta":1881953,"kb_avail":88105968,"statfs":{"total":96636764160,"available":90220511232,"internally_reserved":0,"allocated":4488634368,"data_stored":5234507803,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1927119986},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,4,6,8,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:27:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.59999999999999998,"15min":0.53300000000000003},"min":{"1min":0.191,"5min":0.186,"15min":0.183},"max":{"1min":2.855,"5min":6.4669999999999996,"15min":6.8129999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.63600000000000001,"15min":0.54500000000000004},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.298,"5min":6.5890000000000004,"15min":6.7110000000000003},"last":0.29599999999999999}]},{"osd":2,"last update":"Sat Oct 21 05:27:21 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.41299999999999998,"15min":0.42699999999999999},"min":{"1min":0.22800000000000001,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":0.57899999999999996,"5min":2.4889999999999999,"15min":8.7669999999999995},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.49299999999999999,"15min":0.42499999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.3399999999999999,"5min":3.3399999999999999,"15min":4.1920000000000002},"last":0.32200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739409,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5881356,"kb_used_data":4276996,"kb_used_omap":504,"kb_used_meta":1603847,"kb_avail":88490484,"statfs":{"total":96636764160,"available":90614255616,"internally_reserved":0,"allocated":4379643904,"data_stored":5080505324,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1642339705},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.82299999999999995,"15min":0.59799999999999998},"min":{"1min":0.223,"5min":0.19400000000000001,"15min":0.17899999999999999},"max":{"1min":11.164,"5min":11.164,"15min":15.727},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.81999999999999995,"15min":0.63200000000000001},"min":{"1min":0.24199999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":2.282,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:27:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.67500000000000004,"15min":0.59199999999999997},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22},"max":{"1min":0.47699999999999998,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.63400000000000001,"15min":0.63600000000000001},"min":{"1min":0.27500000000000002,"5min":0.25,"15min":0.24199999999999999},"max":{"1min":0.47899999999999998,"5min":5.6120000000000001,"15min":10.967000000000001},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4379164672,"data_stored":5080100864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4488028160,"data_stored":5234077715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8871018496,"data_stored":10318004243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:27:59.810 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:27:59.810 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:28:05.571 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:28:05.565+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:28:19.811 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:28:20.381 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:28:20.382 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:28:20.436 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2700,"stamp":"2023-10-21T05:28:20.339106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28080929921,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2197060,"num_read_kb":8601977,"num_write":2173805,"num_write_kb":8694463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80045,"ondisk_log_size":80045,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22040092,"kb_used_data":17386172,"kb_used_omap":1985,"kb_used_meta":4651902,"kb_avail":261075428,"statfs":{"total":289910292480,"available":267341238272,"internally_reserved":0,"allocated":17803440128,"data_stored":20696617936,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2033642,"internal_metadata":4763547670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,5,11,11,34,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15962112,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4737,"num_read_kb":18864,"num_write":4608,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002066"},"pg_stats":[{"pgid":"2.7","version":"96'15164","reported_seq":20726,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:16.409879+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:28:16.409879+0000","last_peered":"2023-10-21T05:28:16.409879+0000","last_clean":"2023-10-21T05:28:16.409879+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:28:16.409879+0000","last_undegraded":"2023-10-21T05:28:16.409879+0000","last_fullsized":"2023-10-21T05:28:16.409879+0000","mapping_epoch":13,"log_start":"80'5100","ondisk_log_start":"80'5100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7731","reported_seq":11557,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:16.395118+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:28:16.395118+0000","last_peered":"2023-10-21T05:28:16.395118+0000","last_clean":"2023-10-21T05:28:16.395118+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:28:16.395118+0000","last_undegraded":"2023-10-21T05:28:16.395118+0000","last_fullsized":"2023-10-21T05:28:16.395118+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7731,"log_dups_size":0,"ondisk_log_size":7731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3874","reported_seq":5433,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:15.432534+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:28:15.432534+0000","last_peered":"2023-10-21T05:28:15.432534+0000","last_clean":"2023-10-21T05:28:15.432534+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:28:15.432534+0000","last_undegraded":"2023-10-21T05:28:15.432534+0000","last_fullsized":"2023-10-21T05:28:15.432534+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3874,"log_dups_size":0,"ondisk_log_size":3874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7675","reported_seq":10911,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:16.392230+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:28:16.392230+0000","last_peered":"2023-10-21T05:28:16.392230+0000","last_clean":"2023-10-21T05:28:16.392230+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:28:16.392230+0000","last_undegraded":"2023-10-21T05:28:16.392230+0000","last_fullsized":"2023-10-21T05:28:16.392230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7675,"log_dups_size":0,"ondisk_log_size":7675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'555713","reported_seq":1106115,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:16.833967+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:28:16.833967+0000","last_peered":"2023-10-21T05:28:16.833967+0000","last_clean":"2023-10-21T05:28:16.833967+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:28:16.833967+0000","last_undegraded":"2023-10-21T05:28:16.833967+0000","last_fullsized":"2023-10-21T05:28:16.833967+0000","mapping_epoch":16,"log_start":"96'545618","ondisk_log_start":"96'545618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7111880704,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":551450,"num_read_kb":2174104,"num_write":552222,"num_write_kb":2208888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":230,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:26.000205+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:25:26.000205+0000","last_peered":"2023-10-21T05:25:26.000205+0000","last_clean":"2023-10-21T05:25:26.000205+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:25:26.000205+0000","last_undegraded":"2023-10-21T05:25:26.000205+0000","last_fullsized":"2023-10-21T05:25:26.000205+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'560585","reported_seq":1119036,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:18.899439+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:28:18.899439+0000","last_peered":"2023-10-21T05:28:18.899439+0000","last_clean":"2023-10-21T05:28:18.899439+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:28:18.899439+0000","last_undegraded":"2023-10-21T05:28:18.899439+0000","last_fullsized":"2023-10-21T05:28:18.899439+0000","mapping_epoch":16,"log_start":"96'550519","ondisk_log_start":"96'550519","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7237242880,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":559570,"num_read_kb":2205884,"num_write":556949,"num_write_kb":2227796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4321,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:15.542581+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:28:15.542581+0000","last_peered":"2023-10-21T05:28:15.542581+0000","last_clean":"2023-10-21T05:28:15.542581+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:28:15.542581+0000","last_undegraded":"2023-10-21T05:28:15.542581+0000","last_fullsized":"2023-10-21T05:28:15.542581+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'523804","reported_seq":1046305,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:28:16.824077+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:28:16.824077+0000","last_peered":"2023-10-21T05:28:16.824077+0000","last_clean":"2023-10-21T05:28:16.824077+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:28:16.824077+0000","last_undegraded":"2023-10-21T05:28:16.824077+0000","last_fullsized":"2023-10-21T05:28:16.824077+0000","mapping_epoch":16,"log_start":"96'513720","ondisk_log_start":"96'513720","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2105,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 453s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6694633472,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":523158,"num_read_kb":2062288,"num_write":520463,"num_write_kb":2081852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4257,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:15.434654+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:28:15.434654+0000","last_peered":"2023-10-21T05:28:15.434654+0000","last_clean":"2023-10-21T05:28:15.434654+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:28:15.434654+0000","last_undegraded":"2023-10-21T05:28:15.434654+0000","last_fullsized":"2023-10-21T05:28:15.434654+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'546746","reported_seq":1089602,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:28:18.902631+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:28:18.902631+0000","last_peered":"2023-10-21T05:28:18.902631+0000","last_clean":"2023-10-21T05:28:18.902631+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:28:18.902631+0000","last_undegraded":"2023-10-21T05:28:18.902631+0000","last_fullsized":"2023-10-21T05:28:18.902631+0000","mapping_epoch":16,"log_start":"96'536727","ondisk_log_start":"96'536727","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2603,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 705s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":7036579859,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":543794,"num_read_kb":2143380,"num_write":543433,"num_write_kb":2173726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18920","reported_seq":25964,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:16.482651+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:28:16.482651+0000","last_peered":"2023-10-21T05:28:16.482651+0000","last_clean":"2023-10-21T05:28:16.482651+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:28:16.482651+0000","last_undegraded":"2023-10-21T05:28:16.482651+0000","last_fullsized":"2023-10-21T05:28:16.482651+0000","mapping_epoch":13,"log_start":"80'8900","ondisk_log_start":"80'8900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":276,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:02.933153+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:26:02.933153+0000","last_peered":"2023-10-21T05:26:02.933153+0000","last_clean":"2023-10-21T05:26:02.933153+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:26:02.933153+0000","last_undegraded":"2023-10-21T05:26:02.933153+0000","last_fullsized":"2023-10-21T05:26:02.933153+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28080336915,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2177972,"num_read_kb":8585656,"num_write":2173067,"num_write_kb":8692262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17801904128,"data_stored":20695875622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40264,"ondisk_log_size":40264,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39682,"ondisk_log_size":39682,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739413,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9786464,"kb_used_data":8692952,"kb_used_omap":1002,"kb_used_meta":1092501,"kb_avail":84585376,"statfs":{"total":96636764160,"available":86615425024,"internally_reserved":0,"allocated":8901582848,"data_stored":10348025801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":1118721579},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,5,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:27:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60899999999999999,"15min":0.59899999999999998},"min":{"1min":0.27800000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.2440000000000002,"5min":11.939,"15min":11.939},"last":0.249},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.622,"15min":0.58099999999999996},"min":{"1min":0.20899999999999999,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":3.2570000000000001,"5min":7.6029999999999998,"15min":8.8510000000000009},"last":0.35499999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:27:17 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.58799999999999997,"15min":0.56599999999999995},"min":{"1min":0.21199999999999999,"5min":0.20599999999999999,"15min":0.188},"max":{"1min":4.1550000000000002,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.48299999999999998,"15min":0.55700000000000005},"min":{"1min":0.23000000000000001,"5min":0.22700000000000001,"15min":0.19800000000000001},"max":{"1min":4.1680000000000001,"5min":4.1680000000000001,"15min":8.4030000000000005},"last":0.46700000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739410,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6303976,"kb_used_data":4402076,"kb_used_omap":478,"kb_used_meta":1901409,"kb_avail":88067864,"statfs":{"total":96636764160,"available":90181492736,"internally_reserved":0,"allocated":4507725824,"data_stored":5253599259,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":1947042930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,2,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:27:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.59999999999999998,"15min":0.53300000000000003},"min":{"1min":0.191,"5min":0.186,"15min":0.183},"max":{"1min":2.855,"5min":6.4669999999999996,"15min":6.8129999999999997},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.63600000000000001,"15min":0.54500000000000004},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.298,"5min":6.5890000000000004,"15min":6.7110000000000003},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:27:21 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.41299999999999998,"15min":0.42699999999999999},"min":{"1min":0.22800000000000001,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":0.57899999999999996,"5min":2.4889999999999999,"15min":8.7669999999999995},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.49299999999999999,"15min":0.42499999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.3399999999999999,"5min":3.3399999999999999,"15min":4.1920000000000002},"last":0.47599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739413,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5949652,"kb_used_data":4291144,"kb_used_omap":504,"kb_used_meta":1657991,"kb_avail":88422188,"statfs":{"total":96636764160,"available":90544320512,"internally_reserved":0,"allocated":4394131456,"data_stored":5094992876,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1697783161},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,4,16],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.82299999999999995,"15min":0.59799999999999998},"min":{"1min":0.223,"5min":0.19400000000000001,"15min":0.17899999999999999},"max":{"1min":11.164,"5min":11.164,"15min":15.727},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.81999999999999995,"15min":0.63200000000000001},"min":{"1min":0.24199999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":2.282,"5min":6.0529999999999999,"15min":15.757999999999999},"last":0.318}]},{"osd":2,"last update":"Sat Oct 21 05:27:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.67500000000000004,"15min":0.59199999999999997},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22},"max":{"1min":0.47699999999999998,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.372},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.63400000000000001,"15min":0.63600000000000001},"min":{"1min":0.27500000000000002,"5min":0.25,"15min":0.24199999999999999},"max":{"1min":0.47899999999999998,"5min":5.6120000000000001,"15min":10.967000000000001},"last":0.34599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4393721856,"data_stored":5094658048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4507164672,"data_stored":5253214227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8901017600,"data_stored":10348003347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:28:20.438 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:28:20.439 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:28:40.440 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:28:40.981 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:28:40.982 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:28:41.011 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2710,"stamp":"2023-10-21T05:28:40.342529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28106726529,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2204459,"num_read_kb":8631465,"num_write":2181214,"num_write_kb":8724099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79992,"ondisk_log_size":79992,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22249392,"kb_used_data":17445524,"kb_used_omap":1985,"kb_used_meta":4801854,"kb_avail":260866128,"statfs":{"total":289910292480,"available":267126915072,"internally_reserved":0,"allocated":17864216576,"data_stored":20757394384,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2033642,"internal_metadata":4917098518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,7,9,11,18,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16334848,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4686,"num_read_kb":18664,"num_write":4647,"num_write_kb":18588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002045"},"pg_stats":[{"pgid":"2.7","version":"96'15216","reported_seq":20794,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:37.456671+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:28:37.456671+0000","last_peered":"2023-10-21T05:28:37.456671+0000","last_clean":"2023-10-21T05:28:37.456671+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:28:37.456671+0000","last_undegraded":"2023-10-21T05:28:37.456671+0000","last_fullsized":"2023-10-21T05:28:37.456671+0000","mapping_epoch":13,"log_start":"80'5200","ondisk_log_start":"80'5200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7757","reported_seq":11592,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:37.440232+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:28:37.440232+0000","last_peered":"2023-10-21T05:28:37.440232+0000","last_clean":"2023-10-21T05:28:37.440232+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:28:37.440232+0000","last_undegraded":"2023-10-21T05:28:37.440232+0000","last_fullsized":"2023-10-21T05:28:37.440232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7757,"log_dups_size":0,"ondisk_log_size":7757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3887","reported_seq":5450,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:36.024737+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:28:36.024737+0000","last_peered":"2023-10-21T05:28:36.024737+0000","last_clean":"2023-10-21T05:28:36.024737+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:28:36.024737+0000","last_undegraded":"2023-10-21T05:28:36.024737+0000","last_fullsized":"2023-10-21T05:28:36.024737+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3887,"log_dups_size":0,"ondisk_log_size":3887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7701","reported_seq":10945,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:37.440088+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:28:37.440088+0000","last_peered":"2023-10-21T05:28:37.440088+0000","last_clean":"2023-10-21T05:28:37.440088+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:28:37.440088+0000","last_undegraded":"2023-10-21T05:28:37.440088+0000","last_fullsized":"2023-10-21T05:28:37.440088+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7701,"log_dups_size":0,"ondisk_log_size":7701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'557641","reported_seq":1110019,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:36.928880+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:28:36.928880+0000","last_peered":"2023-10-21T05:28:36.928880+0000","last_clean":"2023-10-21T05:28:36.928880+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:28:36.928880+0000","last_undegraded":"2023-10-21T05:28:36.928880+0000","last_fullsized":"2023-10-21T05:28:36.928880+0000","mapping_epoch":16,"log_start":"96'547618","ondisk_log_start":"96'547618","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7118524416,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":553423,"num_read_kb":2181952,"num_write":554150,"num_write_kb":2216600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":230,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:26.000205+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:25:26.000205+0000","last_peered":"2023-10-21T05:25:26.000205+0000","last_clean":"2023-10-21T05:25:26.000205+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:25:26.000205+0000","last_undegraded":"2023-10-21T05:25:26.000205+0000","last_fullsized":"2023-10-21T05:25:26.000205+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'562439","reported_seq":1122781,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:38.954973+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:28:38.954973+0000","last_peered":"2023-10-21T05:28:38.954973+0000","last_clean":"2023-10-21T05:28:38.954973+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:28:38.954973+0000","last_undegraded":"2023-10-21T05:28:38.954973+0000","last_fullsized":"2023-10-21T05:28:38.954973+0000","mapping_epoch":16,"log_start":"96'552419","ondisk_log_start":"96'552419","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7243780096,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":561460,"num_read_kb":2213428,"num_write":558803,"num_write_kb":2235212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4333,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:36.017899+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:28:36.017899+0000","last_peered":"2023-10-21T05:28:36.017899+0000","last_clean":"2023-10-21T05:28:36.017899+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:28:36.017899+0000","last_undegraded":"2023-10-21T05:28:36.017899+0000","last_fullsized":"2023-10-21T05:28:36.017899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'525598","reported_seq":1049875,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:28:36.925711+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:28:36.925711+0000","last_peered":"2023-10-21T05:28:36.925711+0000","last_clean":"2023-10-21T05:28:36.925711+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:28:36.925711+0000","last_undegraded":"2023-10-21T05:28:36.925711+0000","last_fullsized":"2023-10-21T05:28:36.925711+0000","mapping_epoch":16,"log_start":"96'515520","ondisk_log_start":"96'515520","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2186,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 474s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6700597248,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":524925,"num_read_kb":2069340,"num_write":522254,"num_write_kb":2089016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4269,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:35.925741+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:28:35.925741+0000","last_peered":"2023-10-21T05:28:35.925741+0000","last_clean":"2023-10-21T05:28:35.925741+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:28:35.925741+0000","last_undegraded":"2023-10-21T05:28:35.925741+0000","last_fullsized":"2023-10-21T05:28:35.925741+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'548635","reported_seq":1093270,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:28:38.970143+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:28:38.970143+0000","last_peered":"2023-10-21T05:28:38.970143+0000","last_clean":"2023-10-21T05:28:38.970143+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:28:38.970143+0000","last_undegraded":"2023-10-21T05:28:38.970143+0000","last_fullsized":"2023-10-21T05:28:38.970143+0000","mapping_epoch":16,"log_start":"96'538627","ondisk_log_start":"96'538627","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2681,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 725s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":7043231763,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":545563,"num_read_kb":2150424,"num_write":545269,"num_write_kb":2181070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'18985","reported_seq":26049,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:37.515957+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:28:37.515957+0000","last_peered":"2023-10-21T05:28:37.515957+0000","last_clean":"2023-10-21T05:28:37.515957+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:28:37.515957+0000","last_undegraded":"2023-10-21T05:28:37.515957+0000","last_fullsized":"2023-10-21T05:28:37.515957+0000","mapping_epoch":13,"log_start":"80'8900","ondisk_log_start":"80'8900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":276,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:02.933153+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:26:02.933153+0000","last_peered":"2023-10-21T05:26:02.933153+0000","last_clean":"2023-10-21T05:26:02.933153+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:26:02.933153+0000","last_undegraded":"2023-10-21T05:26:02.933153+0000","last_fullsized":"2023-10-21T05:26:02.933153+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28106133523,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2185371,"num_read_kb":8615144,"num_write":2180476,"num_write_kb":8721898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17862635520,"data_stored":20756607014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40129,"ondisk_log_size":40129,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39764,"ondisk_log_size":39764,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739417,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9879892,"kb_used_data":8722504,"kb_used_omap":1002,"kb_used_meta":1156373,"kb_avail":84491948,"statfs":{"total":96636764160,"available":86519754752,"internally_reserved":0,"allocated":8931844096,"data_stored":10378287049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":1184126507},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:20 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.68999999999999995,"15min":0.61899999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.0060000000000002,"5min":11.939,"15min":11.939},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.1910000000000001,"5min":0.70199999999999996,"15min":0.62},"min":{"1min":0.215,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":9.5790000000000006,"5min":9.5790000000000006,"15min":9.5790000000000006},"last":0.38}]},{"osd":1,"last update":"Sat Oct 21 05:28:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.51700000000000002,"15min":0.56299999999999994},"min":{"1min":0.22500000000000001,"5min":0.20599999999999999,"15min":0.188},"max":{"1min":0.498,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.25,"5min":0.22700000000000001,"15min":0.19800000000000001},"max":{"1min":5.9640000000000004,"5min":5.9640000000000004,"15min":8.4030000000000005},"last":0.33700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739414,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6373020,"kb_used_data":4416980,"kb_used_omap":478,"kb_used_meta":1955553,"kb_avail":87998820,"statfs":{"total":96636764160,"available":90110791680,"internally_reserved":0,"allocated":4522987520,"data_stored":5268860955,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":490382,"internal_metadata":2002486386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,3,4,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.57899999999999996,"15min":0.52600000000000002},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.183},"max":{"1min":0.54100000000000004,"5min":6.4669999999999996,"15min":6.8129999999999997},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.315,"5min":0.61199999999999999,"15min":0.54000000000000004},"min":{"1min":0.218,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.47299999999999998,"5min":6.5890000000000004,"15min":6.7110000000000003},"last":0.29199999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42999999999999999,"15min":0.40100000000000002},"min":{"1min":0.222,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":1.9379999999999999,"5min":2.4889999999999999,"15min":4.3259999999999996},"last":0.253},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.51000000000000001,"15min":0.42899999999999999},"min":{"1min":0.26000000000000001,"5min":0.22,"15min":0.22},"max":{"1min":1.9670000000000001,"5min":3.3399999999999999,"15min":4.1920000000000002},"last":0.33300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739417,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5996480,"kb_used_data":4306040,"kb_used_omap":504,"kb_used_meta":1689927,"kb_avail":88375360,"statfs":{"total":96636764160,"available":90496368640,"internally_reserved":0,"allocated":4409384960,"data_stored":5110246380,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1730485625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,3,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:28:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.68899999999999995,"15min":0.59999999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":1.6160000000000001,"5min":11.164,"15min":15.727},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.60099999999999998,"15min":0.63500000000000001},"min":{"1min":0.23599999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.712,"5min":5.9630000000000001,"15min":15.757999999999999},"last":0.26400000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:28:29 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.56599999999999995,"15min":0.59299999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.22},"max":{"1min":1.7889999999999999,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.48899999999999999,"15min":0.63500000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.8879999999999999,"5min":5.6120000000000001,"15min":10.967000000000001},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4408963072,"data_stored":5109899264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4522274816,"data_stored":5268324371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8931397632,"data_stored":10378383379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:28:41.013 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:28:41.013 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:29:01.015 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:29:01.877 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:29:01.878 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:29:01.910 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2720,"stamp":"2023-10-21T05:29:00.347624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28132564097,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2211754,"num_read_kb":8660553,"num_write":2188620,"num_write_kb":8753723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80196,"ondisk_log_size":80196,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21906000,"kb_used_data":17504696,"kb_used_omap":1987,"kb_used_meta":4399292,"kb_avail":261209520,"statfs":{"total":289910292480,"available":267478548480,"internally_reserved":0,"allocated":17924808704,"data_stored":20817986512,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2035321,"internal_metadata":4504875399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,4,9,11,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16306176,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4559,"num_read_kb":18164,"num_write":4657,"num_write_kb":18628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003703"},"pg_stats":[{"pgid":"2.7","version":"96'15260","reported_seq":20855,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:57.712113+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:28:57.712113+0000","last_peered":"2023-10-21T05:28:57.712113+0000","last_clean":"2023-10-21T05:28:57.712113+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:28:57.712113+0000","last_undegraded":"2023-10-21T05:28:57.712113+0000","last_fullsized":"2023-10-21T05:28:57.712113+0000","mapping_epoch":13,"log_start":"80'5200","ondisk_log_start":"80'5200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7779","reported_seq":11622,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:57.682810+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:28:57.682810+0000","last_peered":"2023-10-21T05:28:57.682810+0000","last_clean":"2023-10-21T05:28:57.682810+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:28:57.682810+0000","last_undegraded":"2023-10-21T05:28:57.682810+0000","last_fullsized":"2023-10-21T05:28:57.682810+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7779,"log_dups_size":0,"ondisk_log_size":7779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3898","reported_seq":5465,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:56.392649+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:28:56.392649+0000","last_peered":"2023-10-21T05:28:56.392649+0000","last_clean":"2023-10-21T05:28:56.392649+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:28:56.392649+0000","last_undegraded":"2023-10-21T05:28:56.392649+0000","last_fullsized":"2023-10-21T05:28:56.392649+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3898,"log_dups_size":0,"ondisk_log_size":3898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7723","reported_seq":10975,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:57.685114+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:28:57.685114+0000","last_peered":"2023-10-21T05:28:57.685114+0000","last_clean":"2023-10-21T05:28:57.685114+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:28:57.685114+0000","last_undegraded":"2023-10-21T05:28:57.685114+0000","last_fullsized":"2023-10-21T05:28:57.685114+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7723,"log_dups_size":0,"ondisk_log_size":7723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'559601","reported_seq":1113828,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:57.115495+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:28:57.115495+0000","last_peered":"2023-10-21T05:28:57.115495+0000","last_clean":"2023-10-21T05:28:57.115495+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:28:57.115495+0000","last_undegraded":"2023-10-21T05:28:57.115495+0000","last_fullsized":"2023-10-21T05:28:57.115495+0000","mapping_epoch":16,"log_start":"96'549518","ondisk_log_start":"96'549518","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7125262336,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":555273,"num_read_kb":2189332,"num_write":556110,"num_write_kb":2224440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":230,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:26.000205+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:25:26.000205+0000","last_peered":"2023-10-21T05:25:26.000205+0000","last_clean":"2023-10-21T05:25:26.000205+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:25:26.000205+0000","last_undegraded":"2023-10-21T05:25:26.000205+0000","last_fullsized":"2023-10-21T05:25:26.000205+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'564310","reported_seq":1126525,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:59.032918+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:28:59.032918+0000","last_peered":"2023-10-21T05:28:59.032918+0000","last_clean":"2023-10-21T05:28:59.032918+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:28:59.032918+0000","last_undegraded":"2023-10-21T05:28:59.032918+0000","last_fullsized":"2023-10-21T05:28:59.032918+0000","mapping_epoch":16,"log_start":"96'554219","ondisk_log_start":"96'554219","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7250444288,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":563332,"num_read_kb":2220872,"num_write":560674,"num_write_kb":2242696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4345,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:56.434279+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:28:56.434279+0000","last_peered":"2023-10-21T05:28:56.434279+0000","last_clean":"2023-10-21T05:28:56.434279+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:28:56.434279+0000","last_undegraded":"2023-10-21T05:28:56.434279+0000","last_fullsized":"2023-10-21T05:28:56.434279+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'527331","reported_seq":1053356,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:28:57.106648+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:28:57.106648+0000","last_peered":"2023-10-21T05:28:57.106648+0000","last_clean":"2023-10-21T05:28:57.106648+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:28:57.106648+0000","last_undegraded":"2023-10-21T05:28:57.106648+0000","last_fullsized":"2023-10-21T05:28:57.106648+0000","mapping_epoch":16,"log_start":"96'517320","ondisk_log_start":"96'517320","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2267,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 494s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6706548736,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":526661,"num_read_kb":2076272,"num_write":523985,"num_write_kb":2095940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4281,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:56.399288+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:28:56.399288+0000","last_peered":"2023-10-21T05:28:56.399288+0000","last_clean":"2023-10-21T05:28:56.399288+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:28:56.399288+0000","last_undegraded":"2023-10-21T05:28:56.399288+0000","last_fullsized":"2023-10-21T05:28:56.399288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'550521","reported_seq":1096999,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:28:59.025332+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:28:59.025332+0000","last_peered":"2023-10-21T05:28:59.025332+0000","last_clean":"2023-10-21T05:28:59.025332+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:28:59.025332+0000","last_undegraded":"2023-10-21T05:28:59.025332+0000","last_fullsized":"2023-10-21T05:28:59.025332+0000","mapping_epoch":16,"log_start":"96'540427","ondisk_log_start":"96'540427","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2752,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 745s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":7049715731,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":547400,"num_read_kb":2157756,"num_write":547113,"num_write_kb":2188446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'19040","reported_seq":26124,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:28:57.758465+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:28:57.758465+0000","last_peered":"2023-10-21T05:28:57.758465+0000","last_clean":"2023-10-21T05:28:57.758465+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:28:57.758465+0000","last_undegraded":"2023-10-21T05:28:57.758465+0000","last_fullsized":"2023-10-21T05:28:57.758465+0000","mapping_epoch":13,"log_start":"80'9000","ondisk_log_start":"80'9000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":276,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:02.933153+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:26:02.933153+0000","last_peered":"2023-10-21T05:26:02.933153+0000","last_clean":"2023-10-21T05:26:02.933153+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:26:02.933153+0000","last_undegraded":"2023-10-21T05:26:02.933153+0000","last_fullsized":"2023-10-21T05:26:02.933153+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28131971091,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2192666,"num_read_kb":8644232,"num_write":2187882,"num_write_kb":8751522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17923280896,"data_stored":20817252390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40279,"ondisk_log_size":40279,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39818,"ondisk_log_size":39818,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739421,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9993828,"kb_used_data":8751772,"kb_used_omap":1002,"kb_used_meta":1241045,"kb_avail":84378012,"statfs":{"total":96636764160,"available":86403084288,"internally_reserved":0,"allocated":8961814528,"data_stored":10408257481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":1270830635},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:20 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.68999999999999995,"15min":0.61899999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.0060000000000002,"5min":11.939,"15min":11.939},"last":0.245},{"interface":"front","average":{"1min":1.1910000000000001,"5min":0.70199999999999996,"15min":0.62},"min":{"1min":0.215,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":9.5790000000000006,"5min":9.5790000000000006,"15min":9.5790000000000006},"last":0.26900000000000002}]},{"osd":1,"last update":"Sat Oct 21 05:28:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.51700000000000002,"15min":0.56299999999999994},"min":{"1min":0.22500000000000001,"5min":0.20599999999999999,"15min":0.188},"max":{"1min":0.498,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.25,"5min":0.22700000000000001,"15min":0.19800000000000001},"max":{"1min":5.9640000000000004,"5min":5.9640000000000004,"15min":8.4030000000000005},"last":0.309}]}]},{"osd":1,"up_from":8,"seq":34359739418,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5864308,"kb_used_data":4431596,"kb_used_omap":480,"kb_used_meta":1432223,"kb_avail":88507532,"statfs":{"total":96636764160,"available":90631712768,"internally_reserved":0,"allocated":4537954304,"data_stored":5283827739,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":492061,"internal_metadata":1466596835},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,4,0,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.57899999999999996,"15min":0.52600000000000002},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.183},"max":{"1min":0.54100000000000004,"5min":6.4669999999999996,"15min":6.8129999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.315,"5min":0.61199999999999999,"15min":0.54000000000000004},"min":{"1min":0.218,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.47299999999999998,"5min":6.5890000000000004,"15min":6.7110000000000003},"last":0.312}]},{"osd":2,"last update":"Sat Oct 21 05:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42999999999999999,"15min":0.40100000000000002},"min":{"1min":0.222,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":1.9379999999999999,"5min":2.4889999999999999,"15min":4.3259999999999996},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.51000000000000001,"15min":0.42899999999999999},"min":{"1min":0.26000000000000001,"5min":0.22,"15min":0.22},"max":{"1min":1.9670000000000001,"5min":3.3399999999999999,"15min":4.1920000000000002},"last":0.35799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739421,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6047864,"kb_used_data":4321328,"kb_used_omap":504,"kb_used_meta":1726023,"kb_avail":88323976,"statfs":{"total":96636764160,"available":90443751424,"internally_reserved":0,"allocated":4425039872,"data_stored":5125901292,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1767447929},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:28:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.68899999999999995,"15min":0.59999999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":1.6160000000000001,"5min":11.164,"15min":15.727},"last":0.316},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.60099999999999998,"15min":0.63500000000000001},"min":{"1min":0.23599999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.712,"5min":5.9630000000000001,"15min":15.757999999999999},"last":1.899}]},{"osd":2,"last update":"Sat Oct 21 05:28:29 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.56599999999999995,"15min":0.59299999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.22},"max":{"1min":1.7889999999999999,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.48899999999999999,"15min":0.63500000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.8879999999999999,"5min":5.6120000000000001,"15min":10.967000000000001},"last":0.34599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4424069120,"data_stored":5125005312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4537507840,"data_stored":5283557395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8961703936,"data_stored":10408689683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:29:01.913 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:29:01.913 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:29:21.914 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:29:22.437 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:29:22.438 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:29:22.471 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2731,"stamp":"2023-10-21T05:29:22.351415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28161485953,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2220032,"num_read_kb":8693565,"num_write":2196870,"num_write_kb":8786723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80159,"ondisk_log_size":80159,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22194840,"kb_used_data":17566716,"kb_used_omap":1987,"kb_used_meta":4626108,"kb_avail":260920680,"statfs":{"total":289910292480,"available":267182776320,"internally_reserved":0,"allocated":17988317184,"data_stored":20881494992,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2035321,"internal_metadata":4737134983},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,4,10,16,15,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16191488,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4578,"num_read_kb":18252,"num_write":4628,"num_write_kb":18512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002096"},"pg_stats":[{"pgid":"2.7","version":"96'15304","reported_seq":20915,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:17.875385+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:29:17.875385+0000","last_peered":"2023-10-21T05:29:17.875385+0000","last_clean":"2023-10-21T05:29:17.875385+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:29:17.875385+0000","last_undegraded":"2023-10-21T05:29:17.875385+0000","last_fullsized":"2023-10-21T05:29:17.875385+0000","mapping_epoch":13,"log_start":"80'5300","ondisk_log_start":"80'5300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7801","reported_seq":11652,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:17.851611+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:29:17.851611+0000","last_peered":"2023-10-21T05:29:17.851611+0000","last_clean":"2023-10-21T05:29:17.851611+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:29:17.851611+0000","last_undegraded":"2023-10-21T05:29:17.851611+0000","last_fullsized":"2023-10-21T05:29:17.851611+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3909","reported_seq":5480,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:16.872300+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:29:16.872300+0000","last_peered":"2023-10-21T05:29:16.872300+0000","last_clean":"2023-10-21T05:29:16.872300+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:29:16.872300+0000","last_undegraded":"2023-10-21T05:29:16.872300+0000","last_fullsized":"2023-10-21T05:29:16.872300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3909,"log_dups_size":0,"ondisk_log_size":3909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7745","reported_seq":11005,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:17.843066+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:29:17.843066+0000","last_peered":"2023-10-21T05:29:17.843066+0000","last_clean":"2023-10-21T05:29:17.843066+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:29:17.843066+0000","last_undegraded":"2023-10-21T05:29:17.843066+0000","last_fullsized":"2023-10-21T05:29:17.843066+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7745,"log_dups_size":0,"ondisk_log_size":7745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'561923","reported_seq":1118449,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:22.212131+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:29:22.212131+0000","last_peered":"2023-10-21T05:29:22.212131+0000","last_clean":"2023-10-21T05:29:22.212131+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:29:22.212131+0000","last_undegraded":"2023-10-21T05:29:22.212131+0000","last_fullsized":"2023-10-21T05:29:22.212131+0000","mapping_epoch":16,"log_start":"96'551918","ondisk_log_start":"96'551918","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7133347840,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":557573,"num_read_kb":2198492,"num_write":558432,"num_write_kb":2233728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":230,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:25:26.000205+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:25:26.000205+0000","last_peered":"2023-10-21T05:25:26.000205+0000","last_clean":"2023-10-21T05:25:26.000205+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:25:26.000205+0000","last_undegraded":"2023-10-21T05:25:26.000205+0000","last_fullsized":"2023-10-21T05:25:26.000205+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'566196","reported_seq":1130302,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:19.248858+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:29:19.248858+0000","last_peered":"2023-10-21T05:29:19.248858+0000","last_clean":"2023-10-21T05:29:19.248858+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:29:19.248858+0000","last_undegraded":"2023-10-21T05:29:19.248858+0000","last_fullsized":"2023-10-21T05:29:19.248858+0000","mapping_epoch":16,"log_start":"96'556119","ondisk_log_start":"96'556119","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7257219072,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":565220,"num_read_kb":2228408,"num_write":562560,"num_write_kb":2250240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'141","reported_seq":4357,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:16.877673+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:29:16.877673+0000","last_peered":"2023-10-21T05:29:16.877673+0000","last_clean":"2023-10-21T05:29:16.877673+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:29:16.877673+0000","last_undegraded":"2023-10-21T05:29:16.877673+0000","last_fullsized":"2023-10-21T05:29:16.877673+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'529577","reported_seq":1057889,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:29:22.256956+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:29:22.256956+0000","last_peered":"2023-10-21T05:29:22.256956+0000","last_clean":"2023-10-21T05:29:22.256956+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:29:22.256956+0000","last_undegraded":"2023-10-21T05:29:22.256956+0000","last_fullsized":"2023-10-21T05:29:22.256956+0000","mapping_epoch":16,"log_start":"96'519520","ondisk_log_start":"96'519520","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2372,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 519s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6714241024,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":528940,"num_read_kb":2085368,"num_write":526224,"num_write_kb":2104896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'142","reported_seq":4293,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:16.872232+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:29:16.872232+0000","last_peered":"2023-10-21T05:29:16.872232+0000","last_clean":"2023-10-21T05:29:16.872232+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:29:16.872232+0000","last_undegraded":"2023-10-21T05:29:16.872232+0000","last_fullsized":"2023-10-21T05:29:16.872232+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'552376","reported_seq":1100678,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:29:19.238888+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:29:19.238888+0000","last_peered":"2023-10-21T05:29:19.238888+0000","last_clean":"2023-10-21T05:29:19.238888+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:29:19.238888+0000","last_undegraded":"2023-10-21T05:29:19.238888+0000","last_fullsized":"2023-10-21T05:29:19.238888+0000","mapping_epoch":16,"log_start":"96'542327","ondisk_log_start":"96'542327","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2832,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 765s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":7056085011,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":549211,"num_read_kb":2164976,"num_write":548916,"num_write_kb":2195658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'19095","reported_seq":26199,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:17.952108+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:29:17.952108+0000","last_peered":"2023-10-21T05:29:17.952108+0000","last_clean":"2023-10-21T05:29:17.952108+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:29:17.952108+0000","last_undegraded":"2023-10-21T05:29:17.952108+0000","last_fullsized":"2023-10-21T05:29:17.952108+0000","mapping_epoch":13,"log_start":"80'9000","ondisk_log_start":"80'9000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":276,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:02.933153+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:26:02.933153+0000","last_peered":"2023-10-21T05:26:02.933153+0000","last_clean":"2023-10-21T05:26:02.933153+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:26:02.933153+0000","last_undegraded":"2023-10-21T05:26:02.933153+0000","last_fullsized":"2023-10-21T05:26:02.933153+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28160892947,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2200944,"num_read_kb":8677244,"num_write":2196132,"num_write_kb":8784522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17987002368,"data_stored":20880973862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40188,"ondisk_log_size":40188,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39872,"ondisk_log_size":39872,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739425,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10116408,"kb_used_data":8780784,"kb_used_omap":1002,"kb_used_meta":1334613,"kb_avail":84255432,"statfs":{"total":96636764160,"available":86277562368,"internally_reserved":0,"allocated":8991522816,"data_stored":10437965769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":1366644267},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:20 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.68999999999999995,"15min":0.61899999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.0060000000000002,"5min":11.939,"15min":11.939},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.1910000000000001,"5min":0.70199999999999996,"15min":0.62},"min":{"1min":0.215,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":9.5790000000000006,"5min":9.5790000000000006,"15min":9.5790000000000006},"last":0.26300000000000001}]},{"osd":1,"last update":"Sat Oct 21 05:28:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.51700000000000002,"15min":0.56299999999999994},"min":{"1min":0.22500000000000001,"5min":0.20599999999999999,"15min":0.188},"max":{"1min":0.498,"5min":11.901999999999999,"15min":11.901999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.25,"5min":0.22700000000000001,"15min":0.19800000000000001},"max":{"1min":5.9640000000000004,"5min":5.9640000000000004,"15min":8.4030000000000005},"last":0.23499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739422,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5946700,"kb_used_data":4446400,"kb_used_omap":480,"kb_used_meta":1499807,"kb_avail":88425140,"statfs":{"total":96636764160,"available":90547343360,"internally_reserved":0,"allocated":4553113600,"data_stored":5298987035,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":492061,"internal_metadata":1535802851},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,9,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.57899999999999996,"15min":0.52600000000000002},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.183},"max":{"1min":0.54100000000000004,"5min":6.4669999999999996,"15min":6.8129999999999997},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.315,"5min":0.61199999999999999,"15min":0.54000000000000004},"min":{"1min":0.218,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.47299999999999998,"5min":6.5890000000000004,"15min":6.7110000000000003},"last":0.80000000000000004}]},{"osd":2,"last update":"Sat Oct 21 05:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42999999999999999,"15min":0.40100000000000002},"min":{"1min":0.222,"5min":0.221,"15min":0.20399999999999999},"max":{"1min":1.9379999999999999,"5min":2.4889999999999999,"15min":4.3259999999999996},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.51000000000000001,"15min":0.42899999999999999},"min":{"1min":0.26000000000000001,"5min":0.22,"15min":0.22},"max":{"1min":1.9670000000000001,"5min":3.3399999999999999,"15min":4.1920000000000002},"last":0.32700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739426,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6131732,"kb_used_data":4339532,"kb_used_omap":504,"kb_used_meta":1791687,"kb_avail":88240108,"statfs":{"total":96636764160,"available":90357870592,"internally_reserved":0,"allocated":4443680768,"data_stored":5144542188,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1834687865},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,8,2,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:28:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.68899999999999995,"15min":0.59999999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":1.6160000000000001,"5min":11.164,"15min":15.727},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.60099999999999998,"15min":0.63500000000000001},"min":{"1min":0.23599999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.712,"5min":5.9630000000000001,"15min":15.757999999999999},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Oct 21 05:28:29 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.56599999999999995,"15min":0.59299999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.22},"max":{"1min":1.7889999999999999,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.48899999999999999,"15min":0.63500000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.8879999999999999,"5min":5.6120000000000001,"15min":10.967000000000001},"last":0.42899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4442759168,"data_stored":5143695360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4552613888,"data_stored":5298663443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8991629312,"data_stored":10438615059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:29:22.473 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:29:22.474 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-21T05:29:22.963 INFO:teuthology.orchestra.run.smithi133.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-21T05:29:23.016 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:29:43.018 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:29:43.505 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:29:43.505 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:29:43.536 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2741,"stamp":"2023-10-21T05:29:42.354901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28184374401,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2226459,"num_read_kb":8719213,"num_write":2203379,"num_write_kb":8812759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80261,"ondisk_log_size":80261,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22450688,"kb_used_data":17622056,"kb_used_omap":1987,"kb_used_meta":4826620,"kb_avail":260664832,"statfs":{"total":289910292480,"available":266920787968,"internally_reserved":0,"allocated":18044985344,"data_stored":20938163152,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2035321,"internal_metadata":4942459271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,5,9,13,18,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13213696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3568,"num_read_kb":14252,"num_write":3764,"num_write_kb":15056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002020"},"pg_stats":[{"pgid":"2.7","version":"96'15340","reported_seq":20967,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:38.642546+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:29:38.642546+0000","last_peered":"2023-10-21T05:29:38.642546+0000","last_clean":"2023-10-21T05:29:38.642546+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:29:38.642546+0000","last_undegraded":"2023-10-21T05:29:38.642546+0000","last_fullsized":"2023-10-21T05:29:38.642546+0000","mapping_epoch":13,"log_start":"80'5300","ondisk_log_start":"80'5300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7819","reported_seq":11678,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:38.611213+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:29:38.611213+0000","last_peered":"2023-10-21T05:29:38.611213+0000","last_clean":"2023-10-21T05:29:38.611213+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:29:38.611213+0000","last_undegraded":"2023-10-21T05:29:38.611213+0000","last_fullsized":"2023-10-21T05:29:38.611213+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3918","reported_seq":5494,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:38.611125+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:29:38.611125+0000","last_peered":"2023-10-21T05:29:38.611125+0000","last_clean":"2023-10-21T05:29:38.611125+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:29:38.611125+0000","last_undegraded":"2023-10-21T05:29:38.611125+0000","last_fullsized":"2023-10-21T05:29:38.611125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7763","reported_seq":11032,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:38.620070+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:29:38.620070+0000","last_peered":"2023-10-21T05:29:38.620070+0000","last_clean":"2023-10-21T05:29:38.620070+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:29:38.620070+0000","last_undegraded":"2023-10-21T05:29:38.620070+0000","last_fullsized":"2023-10-21T05:29:38.620070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7763,"log_dups_size":0,"ondisk_log_size":7763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'563316","reported_seq":1121179,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:37.312164+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:29:37.312164+0000","last_peered":"2023-10-21T05:29:37.312164+0000","last_clean":"2023-10-21T05:29:37.312164+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:29:37.312164+0000","last_undegraded":"2023-10-21T05:29:37.312164+0000","last_fullsized":"2023-10-21T05:29:37.312164+0000","mapping_epoch":16,"log_start":"96'553218","ondisk_log_start":"96'553218","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7138131968,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":558913,"num_read_kb":2203836,"num_write":559825,"num_write_kb":2239300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":231,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:31.617895+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:27:31.617895+0000","last_peered":"2023-10-21T05:27:31.617895+0000","last_clean":"2023-10-21T05:27:31.617895+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:27:31.617895+0000","last_undegraded":"2023-10-21T05:27:31.617895+0000","last_fullsized":"2023-10-21T05:27:31.617895+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'568100","reported_seq":1134118,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:39.410253+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:29:39.410253+0000","last_peered":"2023-10-21T05:29:39.410253+0000","last_clean":"2023-10-21T05:29:39.410253+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:29:39.410253+0000","last_undegraded":"2023-10-21T05:29:39.410253+0000","last_fullsized":"2023-10-21T05:29:39.410253+0000","mapping_epoch":16,"log_start":"96'558019","ondisk_log_start":"96'558019","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7263916032,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":567132,"num_read_kb":2236044,"num_write":564464,"num_write_kb":2257856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"96'144","reported_seq":4370,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:37.359551+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:29:37.359551+0000","last_peered":"2023-10-21T05:29:37.359551+0000","last_clean":"2023-10-21T05:29:37.359551+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:29:37.359551+0000","last_undegraded":"2023-10-21T05:29:37.359551+0000","last_fullsized":"2023-10-21T05:29:37.359551+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'530966","reported_seq":1060604,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:29:37.305994+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:29:37.305994+0000","last_peered":"2023-10-21T05:29:37.305994+0000","last_clean":"2023-10-21T05:29:37.305994+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:29:37.305994+0000","last_undegraded":"2023-10-21T05:29:37.305994+0000","last_fullsized":"2023-10-21T05:29:37.305994+0000","mapping_epoch":16,"log_start":"96'520920","ondisk_log_start":"96'520920","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2414,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 534s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6719053824,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":530257,"num_read_kb":2090628,"num_write":527611,"num_write_kb":2110444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"96'145","reported_seq":4305,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:37.506598+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:29:37.506598+0000","last_peered":"2023-10-21T05:29:37.506598+0000","last_clean":"2023-10-21T05:29:37.506598+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:29:37.506598+0000","last_undegraded":"2023-10-21T05:29:37.506598+0000","last_fullsized":"2023-10-21T05:29:37.506598+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'554260","reported_seq":1104433,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:29:39.432925+0000","last_change":"2023-10-21T05:16:33.488816+0000","last_active":"2023-10-21T05:29:39.432925+0000","last_peered":"2023-10-21T05:29:39.432925+0000","last_clean":"2023-10-21T05:29:39.432925+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:29:39.432925+0000","last_undegraded":"2023-10-21T05:29:39.432925+0000","last_fullsized":"2023-10-21T05:29:39.432925+0000","mapping_epoch":16,"log_start":"96'544227","ondisk_log_start":"96'544227","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-21T04:01:40.659307+0000","last_clean_scrub_stamp":"2023-10-21T04:01:40.659307+0000","objects_scrubbed":2926,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 785s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":7062679571,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":551069,"num_read_kb":2172384,"num_write":550741,"num_write_kb":2202958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'19140","reported_seq":26265,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:38.663615+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:29:38.663615+0000","last_peered":"2023-10-21T05:29:38.663615+0000","last_clean":"2023-10-21T05:29:38.663615+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:29:38.663615+0000","last_undegraded":"2023-10-21T05:29:38.663615+0000","last_fullsized":"2023-10-21T05:29:38.663615+0000","mapping_epoch":13,"log_start":"80'9100","ondisk_log_start":"80'9100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":276,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:02.933153+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:26:02.933153+0000","last_peered":"2023-10-21T05:26:02.933153+0000","last_clean":"2023-10-21T05:26:02.933153+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:26:02.933153+0000","last_undegraded":"2023-10-21T05:26:02.933153+0000","last_fullsized":"2023-10-21T05:26:02.933153+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28183781395,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2207371,"num_read_kb":8702892,"num_write":2202641,"num_write_kb":8810558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18044071936,"data_stored":20938043430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40258,"ondisk_log_size":40258,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":39904,"ondisk_log_size":39904,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739429,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10261424,"kb_used_data":8810280,"kb_used_omap":1002,"kb_used_meta":1450133,"kb_avail":84110416,"statfs":{"total":96636764160,"available":86129065984,"internally_reserved":0,"allocated":9021726720,"data_stored":10468169673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":1484936747},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:29:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.56999999999999995,"15min":0.59699999999999998},"min":{"1min":0.17999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":1.0009999999999999,"5min":6.0060000000000002,"15min":11.939},"last":0.375},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.58999999999999997,"15min":0.58499999999999996},"min":{"1min":0.215,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":0.81799999999999995,"5min":9.5790000000000006,"15min":9.5790000000000006},"last":0.34699999999999998}]},{"osd":1,"last update":"Sat Oct 21 05:29:25 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.40799999999999997,"15min":0.56100000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":0.69899999999999995,"5min":4.1550000000000002,"15min":11.901999999999999},"last":3.661},{"interface":"front","average":{"1min":0.377,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.23499999999999999,"5min":0.23000000000000001,"15min":0.19800000000000001},"max":{"1min":0.72699999999999998,"5min":5.9640000000000004,"15min":8.4030000000000005},"last":3.6339999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739426,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6011448,"kb_used_data":4461168,"kb_used_omap":480,"kb_used_meta":1549791,"kb_avail":88360392,"statfs":{"total":96636764160,"available":90481041408,"internally_reserved":0,"allocated":4568236032,"data_stored":5314109467,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":492061,"internal_metadata":1586986467},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,4,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.50700000000000001,"15min":0.51800000000000002},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.183},"max":{"1min":0.75900000000000001,"5min":6.4669999999999996,"15min":6.8129999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.56000000000000005,"15min":0.53500000000000003},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.80000000000000004,"5min":6.5890000000000004,"15min":6.7110000000000003},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41899999999999998,"15min":0.40799999999999997},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.21099999999999999},"max":{"1min":2.4590000000000001,"5min":2.4590000000000001,"15min":4.3259999999999996},"last":0.222},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.46100000000000002,"15min":0.439},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.22},"max":{"1min":2.673,"5min":3.3399999999999999,"15min":4.1920000000000002},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739429,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6177816,"kb_used_data":4350608,"kb_used_omap":504,"kb_used_meta":1826695,"kb_avail":88194024,"statfs":{"total":96636764160,"available":90310680576,"internally_reserved":0,"allocated":4455022592,"data_stored":5155884012,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1870536057},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,2,7,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.64900000000000002,"15min":0.60799999999999998},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":2.7000000000000002,"5min":11.164,"15min":15.727},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.61299999999999999,"15min":0.63600000000000001},"min":{"1min":0.23000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.899,"5min":5.9630000000000001,"15min":15.757999999999999},"last":0.27400000000000002}]},{"osd":2,"last update":"Sat Oct 21 05:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.56999999999999995,"15min":0.56899999999999995},"min":{"1min":0.24399999999999999,"5min":0.223,"15min":0.22},"max":{"1min":0.68200000000000005,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.46400000000000002,"15min":0.63100000000000001},"min":{"1min":0.29199999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56200000000000006,"5min":5.6120000000000001,"15min":10.967000000000001},"last":0.309}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4454137856,"data_stored":5155074048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4567883776,"data_stored":5313933331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9022050304,"data_stored":10469036051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:29:43.538 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-21T04:01:40.659307+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=4, tm_min=1, tm_sec=40, tm_wday=5, tm_yday=294, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=21, tm_hour=5, tm_min=7, tm_sec=23, tm_wday=5, tm_yday=294, tm_isdst=0) 2023-10-21T05:29:43.538 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-21T05:30:03.540 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-21T05:30:03.597 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:30:03.589+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b3eaea0 2023-10-21T05:30:03.598 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:30:03.589+0000 7fc58f19b640 -1 reset not still connected to 0x55f03b882270 2023-10-21T05:30:03.608 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:30:03.597+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfc340 2023-10-21T05:30:03.619 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:30:03.613+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bbfcf70 2023-10-21T05:30:03.619 INFO:tasks.ceph.osd.1.smithi133.stderr:2023-10-21T05:30:03.613+0000 7fc58f19b640 -1 reset not still connected to 0x55f03bfe9040 2023-10-21T05:30:04.164 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-21T05:30:04.165 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2023-10-21T05:30:04.202 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":2751,"stamp":"2023-10-21T05:30:02.361180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28210343041,"num_objects":11868,"num_object_clones":6656,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11868,"num_whiteouts":0,"num_read":2233792,"num_read_kb":8748477,"num_write":2210769,"num_write_kb":8842319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80390,"ondisk_log_size":80390,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22687328,"kb_used_data":17683076,"kb_used_omap":1987,"kb_used_meta":5002236,"kb_avail":260428192,"statfs":{"total":289910292480,"available":266678468608,"internally_reserved":0,"allocated":18107469824,"data_stored":21000647632,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":2035321,"internal_metadata":5122290055},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,10,4,8,9,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13103104,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3616,"num_read_kb":14444,"num_write":3707,"num_write_kb":14828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002168"},"pg_stats":[{"pgid":"2.7","version":"96'15400","reported_seq":21043,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:59.408620+0000","last_change":"2023-10-21T05:07:48.706550+0000","last_active":"2023-10-21T05:29:59.408620+0000","last_peered":"2023-10-21T05:29:59.408620+0000","last_clean":"2023-10-21T05:29:59.408620+0000","last_became_active":"2023-10-21T04:01:35.654734+0000","last_became_peered":"2023-10-21T04:01:35.654734+0000","last_unstale":"2023-10-21T05:29:59.408620+0000","last_undegraded":"2023-10-21T05:29:59.408620+0000","last_fullsized":"2023-10-21T05:29:59.408620+0000","mapping_epoch":13,"log_start":"80'5300","ondisk_log_start":"80'5300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'12157","last_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_deep_scrub":"84'12157","last_deep_scrub_stamp":"2023-10-21T05:07:48.706369+0000","last_clean_scrub_stamp":"2023-10-21T05:07:48.706369+0000","objects_scrubbed":9,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:42:29.589536+0000","scrub_duration":0.12206596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3276,"num_read_kb":2763,"num_write":92,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2289,"num_omap_keys":45,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7849","reported_seq":11716,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:59.387237+0000","last_change":"2023-10-21T05:07:44.707194+0000","last_active":"2023-10-21T05:29:59.387237+0000","last_peered":"2023-10-21T05:29:59.387237+0000","last_clean":"2023-10-21T05:29:59.387237+0000","last_became_active":"2023-10-21T04:01:35.652084+0000","last_became_peered":"2023-10-21T04:01:35.652084+0000","last_unstale":"2023-10-21T05:29:59.387237+0000","last_undegraded":"2023-10-21T05:29:59.387237+0000","last_fullsized":"2023-10-21T05:29:59.387237+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6202","last_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_deep_scrub":"84'6202","last_deep_scrub_stamp":"2023-10-21T05:07:44.707007+0000","last_clean_scrub_stamp":"2023-10-21T05:07:44.707007+0000","objects_scrubbed":9,"log_size":7849,"log_dups_size":0,"ondisk_log_size":7849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:08:30.862522+0000","scrub_duration":0.054730184000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3474,"num_read_kb":3149,"num_write":103,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2879,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"96'3933","reported_seq":5513,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:59.263687+0000","last_change":"2023-10-21T05:07:34.697367+0000","last_active":"2023-10-21T05:29:59.263687+0000","last_peered":"2023-10-21T05:29:59.263687+0000","last_clean":"2023-10-21T05:29:59.263687+0000","last_became_active":"2023-10-21T04:01:35.651032+0000","last_became_peered":"2023-10-21T04:01:35.651032+0000","last_unstale":"2023-10-21T05:29:59.263687+0000","last_undegraded":"2023-10-21T05:29:59.263687+0000","last_fullsized":"2023-10-21T05:29:59.263687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3101","last_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_deep_scrub":"84'3101","last_deep_scrub_stamp":"2023-10-21T05:07:34.697160+0000","last_clean_scrub_stamp":"2023-10-21T05:07:34.697160+0000","objects_scrubbed":12,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:56:18.331252+0000","scrub_duration":0.049089779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":850,"num_read_kb":704,"num_write":39,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":518,"num_omap_keys":11,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'7793","reported_seq":11070,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:59.383986+0000","last_change":"2023-10-21T05:07:37.769443+0000","last_active":"2023-10-21T05:29:59.383986+0000","last_peered":"2023-10-21T05:29:59.383986+0000","last_clean":"2023-10-21T05:29:59.383986+0000","last_became_active":"2023-10-21T04:01:35.651880+0000","last_became_peered":"2023-10-21T04:01:35.651880+0000","last_unstale":"2023-10-21T05:29:59.383986+0000","last_undegraded":"2023-10-21T05:29:59.383986+0000","last_fullsized":"2023-10-21T05:29:59.383986+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'6144","last_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_deep_scrub":"84'6144","last_deep_scrub_stamp":"2023-10-21T05:07:37.768766+0000","last_clean_scrub_stamp":"2023-10-21T05:07:37.768766+0000","objects_scrubbed":15,"log_size":7793,"log_dups_size":0,"ondisk_log_size":7793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T08:36:04.170259+0000","scrub_duration":0.095886712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2164,"num_read_kb":1849,"num_write":67,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1782,"num_omap_keys":63,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'565196","reported_seq":1124924,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:57.445431+0000","last_change":"2023-10-21T05:20:41.734990+0000","last_active":"2023-10-21T05:29:57.445431+0000","last_peered":"2023-10-21T05:29:57.445431+0000","last_clean":"2023-10-21T05:29:57.445431+0000","last_became_active":"2023-10-21T04:01:42.548025+0000","last_became_peered":"2023-10-21T04:01:42.548025+0000","last_unstale":"2023-10-21T05:29:57.445431+0000","last_undegraded":"2023-10-21T05:29:57.445431+0000","last_fullsized":"2023-10-21T05:29:57.445431+0000","mapping_epoch":16,"log_start":"96'555118","ondisk_log_start":"96'555118","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'513533","last_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_deep_scrub":"95'513533","last_deep_scrub_stamp":"2023-10-21T05:20:41.734815+0000","last_clean_scrub_stamp":"2023-10-21T05:20:41.734815+0000","objects_scrubbed":2971,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":782,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:04:49.243855+0000","scrub_duration":781.90212121399998,"objects_trimmed":0,"snaptrim_duration":0.000542024,"stat_sum":{"num_bytes":7144751104,"num_objects":2971,"num_object_clones":1672,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":560776,"num_read_kb":2211272,"num_write":561705,"num_write_kb":2246820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"84'35","reported_seq":231,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:27:31.617895+0000","last_change":"2023-10-21T05:07:30.895874+0000","last_active":"2023-10-21T05:27:31.617895+0000","last_peered":"2023-10-21T05:27:31.617895+0000","last_clean":"2023-10-21T05:27:31.617895+0000","last_became_active":"2023-10-21T04:01:35.649632+0000","last_became_peered":"2023-10-21T04:01:35.649632+0000","last_unstale":"2023-10-21T05:27:31.617895+0000","last_undegraded":"2023-10-21T05:27:31.617895+0000","last_fullsized":"2023-10-21T05:27:31.617895+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'35","last_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_deep_scrub":"84'35","last_deep_scrub_stamp":"2023-10-21T05:07:30.895675+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.895675+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:44:51.419962+0000","scrub_duration":0.043479512999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'570013","reported_seq":1137891,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:59.562501+0000","last_change":"2023-10-21T05:21:00.436958+0000","last_active":"2023-10-21T05:29:59.562501+0000","last_peered":"2023-10-21T05:29:59.562501+0000","last_clean":"2023-10-21T05:29:59.562501+0000","last_became_active":"2023-10-21T04:01:42.547989+0000","last_became_peered":"2023-10-21T04:01:42.547989+0000","last_unstale":"2023-10-21T05:29:59.562501+0000","last_undegraded":"2023-10-21T05:29:59.562501+0000","last_fullsized":"2023-10-21T05:29:59.562501+0000","mapping_epoch":16,"log_start":"96'559919","ondisk_log_start":"96'559919","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'519414","last_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_deep_scrub":"96'519414","last_deep_scrub_stamp":"2023-10-21T05:21:00.436775+0000","last_clean_scrub_stamp":"2023-10-21T05:21:00.436775+0000","objects_scrubbed":3055,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":790,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:59:11.428062+0000","scrub_duration":789.790791248,"objects_trimmed":0,"snaptrim_duration":0.001405922,"stat_sum":{"num_bytes":7270793216,"num_objects":3055,"num_object_clones":1742,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":568994,"num_read_kb":2243484,"num_write":566377,"num_write_kb":2265508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"96'144","reported_seq":4382,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:57.847668+0000","last_change":"2023-10-21T05:07:31.352410+0000","last_active":"2023-10-21T05:29:57.847668+0000","last_peered":"2023-10-21T05:29:57.847668+0000","last_clean":"2023-10-21T05:29:57.847668+0000","last_became_active":"2023-10-21T04:01:35.652529+0000","last_became_peered":"2023-10-21T04:01:35.652529+0000","last_unstale":"2023-10-21T05:29:57.847668+0000","last_undegraded":"2023-10-21T05:29:57.847668+0000","last_fullsized":"2023-10-21T05:29:57.847668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'141","last_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_deep_scrub":"84'141","last_deep_scrub_stamp":"2023-10-21T05:07:31.352270+0000","last_clean_scrub_stamp":"2023-10-21T05:07:31.352270+0000","objects_scrubbed":8,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T06:06:10.973656+0000","scrub_duration":0.020747825000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2609,"num_read_kb":2189,"num_write":71,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1988,"num_omap_keys":35,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'532737","reported_seq":1064150,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-21T05:29:57.446019+0000","last_change":"2023-10-21T05:20:42.874582+0000","last_active":"2023-10-21T05:29:57.446019+0000","last_peered":"2023-10-21T05:29:57.446019+0000","last_clean":"2023-10-21T05:29:57.446019+0000","last_became_active":"2023-10-21T04:01:42.548016+0000","last_became_peered":"2023-10-21T04:01:42.548016+0000","last_unstale":"2023-10-21T05:29:57.446019+0000","last_undegraded":"2023-10-21T05:29:57.446019+0000","last_fullsized":"2023-10-21T05:29:57.446019+0000","mapping_epoch":16,"log_start":"96'522720","ondisk_log_start":"96'522720","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'463654","last_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_deep_scrub":"92'463654","last_deep_scrub_stamp":"2023-10-21T05:16:32.733538+0000","last_clean_scrub_stamp":"2023-10-21T05:16:32.733538+0000","objects_scrubbed":2522,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":537,"scrub_schedule":"deep scrubbing for 554s","scrub_duration":536.90250268700004,"objects_trimmed":0,"snaptrim_duration":0.0014384459999999999,"stat_sum":{"num_bytes":6725144576,"num_objects":2779,"num_object_clones":1551,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":532018,"num_read_kb":2097652,"num_write":529377,"num_write_kb":2117508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"96'145","reported_seq":4317,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:57.836471+0000","last_change":"2023-10-21T05:07:30.443384+0000","last_active":"2023-10-21T05:29:57.836471+0000","last_peered":"2023-10-21T05:29:57.836471+0000","last_clean":"2023-10-21T05:29:57.836471+0000","last_became_active":"2023-10-21T04:01:35.649943+0000","last_became_peered":"2023-10-21T04:01:35.649943+0000","last_unstale":"2023-10-21T05:29:57.836471+0000","last_undegraded":"2023-10-21T05:29:57.836471+0000","last_fullsized":"2023-10-21T05:29:57.836471+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'142","last_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_deep_scrub":"84'142","last_deep_scrub_stamp":"2023-10-21T05:07:30.443097+0000","last_clean_scrub_stamp":"2023-10-21T05:07:30.443097+0000","objects_scrubbed":12,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T10:19:02.312350+0000","scrub_duration":0.085775796000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2204,"num_read_kb":1869,"num_write":61,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1120,"num_omap_keys":31,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'556115","reported_seq":1108147,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:59.569312+0000","last_change":"2023-10-21T05:29:54.852176+0000","last_active":"2023-10-21T05:29:59.569312+0000","last_peered":"2023-10-21T05:29:59.569312+0000","last_clean":"2023-10-21T05:29:59.569312+0000","last_became_active":"2023-10-21T04:01:42.548033+0000","last_became_peered":"2023-10-21T04:01:42.548033+0000","last_unstale":"2023-10-21T05:29:59.569312+0000","last_undegraded":"2023-10-21T05:29:59.569312+0000","last_fullsized":"2023-10-21T05:29:59.569312+0000","mapping_epoch":16,"log_start":"96'546027","ondisk_log_start":"96'546027","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'555684","last_scrub_stamp":"2023-10-21T05:29:54.851964+0000","last_deep_scrub":"96'555684","last_deep_scrub_stamp":"2023-10-21T05:29:54.851964+0000","last_clean_scrub_stamp":"2023-10-21T05:29:54.851964+0000","objects_scrubbed":2972,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":802,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T16:20:32.213486+0000","scrub_duration":801.36193237999998,"objects_trimmed":0,"snaptrim_duration":0.001497106,"stat_sum":{"num_bytes":7069061139,"num_objects":2972,"num_object_clones":1691,"num_object_copies":5944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":552916,"num_read_kb":2179748,"num_write":552572,"num_write_kb":2210282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'19215","reported_seq":26360,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:29:59.483395+0000","last_change":"2023-10-21T05:07:32.860116+0000","last_active":"2023-10-21T05:29:59.483395+0000","last_peered":"2023-10-21T05:29:59.483395+0000","last_clean":"2023-10-21T05:29:59.483395+0000","last_became_active":"2023-10-21T04:01:35.649808+0000","last_became_peered":"2023-10-21T04:01:35.649808+0000","last_unstale":"2023-10-21T05:29:59.483395+0000","last_undegraded":"2023-10-21T05:29:59.483395+0000","last_fullsized":"2023-10-21T05:29:59.483395+0000","mapping_epoch":13,"log_start":"80'9200","ondisk_log_start":"80'9200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'15143","last_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_deep_scrub":"84'15143","last_deep_scrub_stamp":"2023-10-21T05:07:32.859924+0000","last_clean_scrub_stamp":"2023-10-21T05:07:32.859924+0000","objects_scrubbed":16,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T12:24:26.418397+0000","scrub_duration":0.21473841900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4334,"num_read_kb":3647,"num_write":141,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3023,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"84'99","reported_seq":276,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-21T05:26:02.933153+0000","last_change":"2023-10-21T05:07:28.876030+0000","last_active":"2023-10-21T05:26:02.933153+0000","last_peered":"2023-10-21T05:26:02.933153+0000","last_clean":"2023-10-21T05:26:02.933153+0000","last_became_active":"2023-10-21T04:01:30.996510+0000","last_became_peered":"2023-10-21T04:01:30.996510+0000","last_unstale":"2023-10-21T05:26:02.933153+0000","last_undegraded":"2023-10-21T05:26:02.933153+0000","last_fullsized":"2023-10-21T05:26:02.933153+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'99","last_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_deep_scrub":"84'99","last_deep_scrub_stamp":"2023-10-21T05:07:28.869501+0000","last_clean_scrub_stamp":"2023-10-21T05:07:28.869501+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-22T13:08:53.767196+0000","scrub_duration":0.061428723999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28209750035,"num_objects":11777,"num_object_clones":6656,"num_object_copies":23554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11777,"num_whiteouts":0,"num_read":2214704,"num_read_kb":8732156,"num_write":2210031,"num_write_kb":8840118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18104737792,"data_stored":20998709286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40277,"ondisk_log_size":40277,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2638,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":18982,"num_read_kb":16230,"num_write":601,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14959,"num_omap_keys":329,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":5276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36027,"internal_metadata":0},"log_size":40014,"ondisk_log_size":40014,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12518,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739433,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10388352,"kb_used_data":8841140,"kb_used_omap":1002,"kb_used_meta":1546197,"kb_avail":83983488,"statfs":{"total":96636764160,"available":85999091712,"internally_reserved":0,"allocated":9053327360,"data_stored":10499770313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1026517,"internal_metadata":1583306283},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:29:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.56999999999999995,"15min":0.59699999999999998},"min":{"1min":0.17999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":1.0009999999999999,"5min":6.0060000000000002,"15min":11.939},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.58999999999999997,"15min":0.58499999999999996},"min":{"1min":0.215,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":0.81799999999999995,"5min":9.5790000000000006,"15min":9.5790000000000006},"last":0.248}]},{"osd":1,"last update":"Sat Oct 21 05:29:25 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.40799999999999997,"15min":0.56100000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":0.69899999999999995,"5min":4.1550000000000002,"15min":11.901999999999999},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.23499999999999999,"5min":0.23000000000000001,"15min":0.19800000000000001},"max":{"1min":0.72699999999999998,"5min":5.9640000000000004,"15min":8.4030000000000005},"last":0.34499999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739430,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6063160,"kb_used_data":4476784,"kb_used_omap":480,"kb_used_meta":1585887,"kb_avail":88308680,"statfs":{"total":96636764160,"available":90428088320,"internally_reserved":0,"allocated":4584226816,"data_stored":5330100251,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":492061,"internal_metadata":1623948771},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 21 05:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.50700000000000001,"15min":0.51800000000000002},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.183},"max":{"1min":0.75900000000000001,"5min":6.4669999999999996,"15min":6.8129999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.56000000000000005,"15min":0.53500000000000003},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.80000000000000004,"5min":6.5890000000000004,"15min":6.7110000000000003},"last":0.26000000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41899999999999998,"15min":0.40799999999999997},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.21099999999999999},"max":{"1min":2.4590000000000001,"5min":2.4590000000000001,"15min":4.3259999999999996},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.46100000000000002,"15min":0.439},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.22},"max":{"1min":2.673,"5min":3.3399999999999999,"15min":4.1920000000000002},"last":0.28599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739433,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6235816,"kb_used_data":4365152,"kb_used_omap":504,"kb_used_meta":1870151,"kb_avail":88136024,"statfs":{"total":96636764160,"available":90251288576,"internally_reserved":0,"allocated":4469915648,"data_stored":5170777068,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":516743,"internal_metadata":1915035001},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,3,1,5,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 21 05:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.64900000000000002,"15min":0.60799999999999998},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":2.7000000000000002,"5min":11.164,"15min":15.727},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.61299999999999999,"15min":0.63600000000000001},"min":{"1min":0.23000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.899,"5min":5.9630000000000001,"15min":15.757999999999999},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Oct 21 05:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.56999999999999995,"15min":0.56899999999999995},"min":{"1min":0.24399999999999999,"5min":0.223,"15min":0.22},"max":{"1min":0.68200000000000005,"5min":11.337999999999999,"15min":11.337999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.46400000000000002,"15min":0.63100000000000001},"min":{"1min":0.29199999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56200000000000006,"5min":5.6120000000000001,"15min":10.967000000000001},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6259,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11588,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15636,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8803,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4469071872,"data_stored":5170008064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4583219200,"data_stored":5329268755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9052446720,"data_stored":10499432467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-21T05:30:04.204 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-21T05:30:04.778 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-21T05:30:04.778 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-21T05:30:04.779 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-21T05:30:04.779 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:30:06.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-21T05:30:10.881 INFO:tasks.ceph.osd.0:Stopped 2023-10-21T05:30:10.881 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-21T05:30:10.882 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:30:12.045 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-21T05:30:12.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-21T05:30:16.983 INFO:tasks.ceph.osd.1:Stopped 2023-10-21T05:30:16.984 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-21T05:30:16.984 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:30:17.348 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-21T05:30:22.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-21T05:30:23.086 INFO:tasks.ceph.osd.2:Stopped 2023-10-21T05:30:23.087 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-21T05:30:23.087 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-21T05:30:23.088 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:30:27.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-10-21T05:30:27.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-21T05:30:29.189 INFO:tasks.ceph.mgr.x:Stopped 2023-10-21T05:30:29.189 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-21T05:30:29.190 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-21T05:30:29.191 INFO:teuthology.orchestra.run:waiting for 300 2023-10-21T05:30:32.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-10-21T05:30:32.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-10-21T05:30:35.292 INFO:tasks.ceph.mon.a:Stopped 2023-10-21T05:30:35.293 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-21T05:30:35.293 DEBUG:teuthology.orchestra.run.smithi133:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-21T05:30:35.395 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi133.front.sepia.ceph.com 2023-10-21T05:30:35.396 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-21T05:30:36.040 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi133.front.sepia.ceph.com 2023-10-21T05:30:36.040 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-21T05:30:36.455 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi133.front.sepia.ceph.com 2023-10-21T05:30:36.456 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-21T05:30:37.014 INFO:tasks.ceph:Archiving mon data... 2023-10-21T05:30:37.016 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433358/data/mon.a.tgz 2023-10-21T05:30:37.016 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2023-10-21T05:30:37.037 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.1OV9W0XetE 2023-10-21T05:30:37.037 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.1OV9W0XetE 2023-10-21T05:30:37.195 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.1OV9W0XetE 2023-10-21T05:30:37.272 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.1OV9W0XetE is 449KB 2023-10-21T05:30:37.332 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.1OV9W0XetE 2023-10-21T05:30:37.339 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-21T05:30:37.339 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-21T05:30:37.440 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-21T05:30:37.440 DEBUG:teuthology.orchestra.run.smithi133:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-21T05:30:37.482 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-21T05:30:37.486 INFO:tasks.ceph:Archiving crash dumps... 2023-10-21T05:30:37.488 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433358/remote/smithi133/crash 2023-10-21T05:30:37.489 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-21T05:30:37.544 INFO:tasks.ceph:Compressing logs... 2023-10-21T05:30:37.545 DEBUG:teuthology.orchestra.run.smithi133:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-21T05:30:37.611 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48900.log 2023-10-21T05:30:37.612 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43287.log 2023-10-21T05:30:37.612 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42434.log 2023-10-21T05:30:37.613 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48900.log.gz 2023-10-21T05:30:37.614 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50634.log 2023-10-21T05:30:37.614 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48867.log 2023-10-21T05:30:37.614 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42543.log/var/log/ceph/ceph-client.admin.50634.log: 2023-10-21T05:30:37.615 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50634.log.gz 2023-10-21T05:30:37.615 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51645.log 2023-10-21T05:30:37.615 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48867.log.gz 2023-10-21T05:30:37.621 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44484.log 2023-10-21T05:30:37.621 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50846.log 2023-10-21T05:30:37.621 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51645.log.gz 2023-10-21T05:30:37.621 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42918.log 2023-10-21T05:30:37.622 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45005.log 2023-10-21T05:30:37.622 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50846.log.gz 2023-10-21T05:30:37.622 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49509.log 2023-10-21T05:30:37.623 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51773.log 2023-10-21T05:30:37.623 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49811.log 2023-10-21T05:30:37.623 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49509.log.gz 2023-10-21T05:30:37.623 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40208.log 2023-10-21T05:30:37.624 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51773.log.gz 2023-10-21T05:30:37.624 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42218.log 2023-10-21T05:30:37.624 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49811.log.gz 2023-10-21T05:30:37.624 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51241.log 2023-10-21T05:30:37.625 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52189.log 2023-10-21T05:30:37.625 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44882.log 2023-10-21T05:30:37.625 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51241.log.gz 2023-10-21T05:30:37.625 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39087.log 2023-10-21T05:30:37.626 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52189.log.gz 2023-10-21T05:30:37.626 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49298.log 2023-10-21T05:30:37.626 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50694.log 2023-10-21T05:30:37.627 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44002.log 2023-10-21T05:30:37.627 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49298.log.gz 2023-10-21T05:30:37.627 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42143.log 2023-10-21T05:30:37.627 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50694.log.gz 2023-10-21T05:30:37.627 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37886.log 2023-10-21T05:30:37.628 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43700.log 2023-10-21T05:30:37.628 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48445.log 2023-10-21T05:30:37.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-10-21T05:30:37.673 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45795.log 2023-10-21T05:30:37.673 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54244.log 2023-10-21T05:30:37.673 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48445.log.gz 2023-10-21T05:30:37.674 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39124.log 2023-10-21T05:30:37.674 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38726.log 2023-10-21T05:30:37.674 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54244.log.gz 2023-10-21T05:30:37.675 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42888.log 2023-10-21T05:30:37.675 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51151.log 2023-10-21T05:30:37.675 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46339.log 2023-10-21T05:30:37.675 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42918.log.gz -- replaced with /var/log/ceph/ceph-client.admin.42434.log.gz 2023-10-21T05:30:37.676 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-10-21T05:30:37.676 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% 0.0% 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39124.log.gz -- replaced with /var/log/ceph/ceph-client.admin.42143.log.gz -- replaced with /var/log/ceph/ceph-client.admin.42888.log.gz -- replaced with /var/log/ceph/ceph-client.admin.42218.log.gz 2023-10-21T05:30:37.676 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-10-21T05:30:37.676 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-10-21T05:30:37.677 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-10-21T05:30:37.677 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43287.log.gz 2023-10-21T05:30:37.677 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42543.log.gz 0.0% 2023-10-21T05:30:37.677 INFO:teuthology.orchestra.run.smithi133.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39087.log.gz 2023-10-21T05:30:37.678 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44002.log.gz 2023-10-21T05:30:37.678 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51151.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43700.log.gz 2023-10-21T05:30:37.678 INFO:teuthology.orchestra.run.smithi133.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44882.log.gz 2023-10-21T05:30:37.678 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48353.log 2023-10-21T05:30:37.679 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45795.log.gz 2023-10-21T05:30:37.679 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45005.log.gz 2023-10-21T05:30:37.679 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38726.log.gz 2023-10-21T05:30:37.679 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44484.log.gz 2023-10-21T05:30:37.680 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51151.log.gz 2023-10-21T05:30:37.680 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40208.log.gz 2023-10-21T05:30:37.680 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43508.log 2023-10-21T05:30:37.680 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46339.log.gz 2023-10-21T05:30:37.681 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45428.log 2023-10-21T05:30:37.681 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48353.log.gz 2023-10-21T05:30:37.681 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44575.log 2023-10-21T05:30:37.682 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43508.log.gz 2023-10-21T05:30:37.682 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45428.log.gz 2023-10-21T05:30:37.682 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41655.log 2023-10-21T05:30:37.682 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53706.log 2023-10-21T05:30:37.683 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44575.log.gz 2023-10-21T05:30:37.683 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41655.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.41655.log.gz --verbose 2023-10-21T05:30:37.683 INFO:teuthology.orchestra.run.smithi133.stderr: -- /var/log/ceph/ceph-client.admin.46738.log 2023-10-21T05:30:37.683 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50876.log 2023-10-21T05:30:37.684 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53706.log.gz 2023-10-21T05:30:37.684 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42573.log 2023-10-21T05:30:37.684 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46738.log.gz 2023-10-21T05:30:37.684 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49022.log 2023-10-21T05:30:37.685 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50876.log.gz 2023-10-21T05:30:37.685 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42573.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.42573.log.gz 2023-10-21T05:30:37.685 INFO:teuthology.orchestra.run.smithi133.stderr: /var/log/ceph/ceph-client.admin.45975.log 2023-10-21T05:30:37.685 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38800.log 2023-10-21T05:30:37.686 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49022.log.gz 2023-10-21T05:30:37.686 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54350.log 2023-10-21T05:30:37.686 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45975.log.gz 2023-10-21T05:30:37.686 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46980.log 2023-10-21T05:30:37.687 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38800.log.gz 2023-10-21T05:30:37.687 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54350.log.gz 2023-10-21T05:30:37.687 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-21T05:30:37.687 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46980.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37886.log.gz 2023-10-21T05:30:37.688 INFO:teuthology.orchestra.run.smithi133.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38709.log 2023-10-21T05:30:37.688 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46980.log.gz 2023-10-21T05:30:37.688 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39622.log 2023-10-21T05:30:37.688 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52269.log 2023-10-21T05:30:37.689 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38709.log.gz 2023-10-21T05:30:37.689 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46890.log 2023-10-21T05:30:37.689 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39622.log.gz 2023-10-21T05:30:37.689 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38962.log 2023-10-21T05:30:37.690 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52269.log.gz 2023-10-21T05:30:37.690 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54298.log 2023-10-21T05:30:37.690 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46890.log.gz 2023-10-21T05:30:37.691 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38962.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.38962.log.gz --verbose 2023-10-21T05:30:37.691 INFO:teuthology.orchestra.run.smithi133.stderr: -- /var/log/ceph/ceph-client.admin.41407.log 2023-10-21T05:30:37.691 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43348.log 2023-10-21T05:30:37.691 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54298.log.gz 2023-10-21T05:30:37.692 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51211.log 2023-10-21T05:30:37.692 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41407.log.gz 2023-10-21T05:30:37.692 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43348.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43348.log.gz 2023-10-21T05:30:37.692 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47773.log 2023-10-21T05:30:37.693 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5/var/log/ceph/ceph-client.admin.51211.log: --verbose -- /var/log/ceph/ceph-client.admin.42678.log 2023-10-21T05:30:37.693 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51211.log.gz 2023-10-21T05:30:37.693 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37937.log 2023-10-21T05:30:37.694 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47773.log.gz 2023-10-21T05:30:37.694 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53624.log 2023-10-21T05:30:37.694 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42678.log.gz 2023-10-21T05:30:37.694 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37790.log 2023-10-21T05:30:37.695 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37937.log.gz 2023-10-21T05:30:37.695 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44029.log 2023-10-21T05:30:37.695 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53624.log.gz 2023-10-21T05:30:37.695 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40878.log 2023-10-21T05:30:37.696 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37790.log.gz 2023-10-21T05:30:37.696 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48686.log 2023-10-21T05:30:37.696 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44029.log.gz 2023-10-21T05:30:37.697 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40460.log 2023-10-21T05:30:37.697 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40878.log.gz 2023-10-21T05:30:37.697 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45368.log 2023-10-21T05:30:37.697 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48686.log.gz 2023-10-21T05:30:37.698 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40974.log 2023-10-21T05:30:37.698 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40460.log.gz 2023-10-21T05:30:37.698 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45368.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45368.log.gz 2023-10-21T05:30:37.699 INFO:teuthology.orchestra.run.smithi133.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46648.log 2023-10-21T05:30:37.699 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37600.log 2023-10-21T05:30:37.699 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40974.log.gz 2023-10-21T05:30:37.699 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49966.log 2023-10-21T05:30:37.700 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46648.log.gz 2023-10-21T05:30:37.700 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37600.log.gz 2023-10-21T05:30:37.700 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38655.log 2023-10-21T05:30:37.701 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44454.log 2023-10-21T05:30:37.701 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49966.log.gz 2023-10-21T05:30:37.701 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41870.log 2023-10-21T05:30:37.701 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38655.log.gz 2023-10-21T05:30:37.702 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44454.log.gz 2023-10-21T05:30:37.702 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49479.log 2023-10-21T05:30:37.702 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43134.log 2023-10-21T05:30:37.702 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41870.log.gz 2023-10-21T05:30:37.703 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46950.log 2023-10-21T05:30:37.703 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49479.log.gz 2023-10-21T05:30:37.703 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45673.log 2023-10-21T05:30:37.704 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43134.log.gz 2023-10-21T05:30:37.704 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46950.log.gz 2023-10-21T05:30:37.704 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52163.log 2023-10-21T05:30:37.704 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53386.log 2023-10-21T05:30:37.705 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45673.log.gz 2023-10-21T05:30:37.705 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38490.log 2023-10-21T05:30:37.705 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52163.log.gz 2023-10-21T05:30:37.706 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41595.log 2023-10-21T05:30:37.706 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53386.log.gz 2023-10-21T05:30:37.706 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38490.log.gz 2023-10-21T05:30:37.706 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43318.log 2023-10-21T05:30:37.707 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34675.log 2023-10-21T05:30:37.707 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41595.log.gz 2023-10-21T05:30:37.707 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47983.log 2023-10-21T05:30:37.707 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43318.log.gz 2023-10-21T05:30:37.708 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.34675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53093.log 2023-10-21T05:30:37.708 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34675.log.gz 2023-10-21T05:30:37.708 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41092.log 2023-10-21T05:30:37.708 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47983.log.gz 2023-10-21T05:30:37.709 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48746.log 2023-10-21T05:30:37.709 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53093.log.gz 2023-10-21T05:30:37.709 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41092.log.gz 2023-10-21T05:30:37.709 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53145.log 2023-10-21T05:30:37.710 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41699.log 2023-10-21T05:30:37.710 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48746.log.gz 2023-10-21T05:30:37.710 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53145.log.gz 2023-10-21T05:30:37.711 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39249.log 2023-10-21T05:30:37.711 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42993.log 2023-10-21T05:30:37.711 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41699.log.gz 2023-10-21T05:30:37.711 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39249.log.gz 2023-10-21T05:30:37.712 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45885.log 2023-10-21T05:30:37.712 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50242.log 2023-10-21T05:30:37.712 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42993.log.gz 2023-10-21T05:30:37.712 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45885.log.gz 2023-10-21T05:30:37.713 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40785.log 2023-10-21T05:30:37.713 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46552.log 2023-10-21T05:30:37.713 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50242.log.gz 2023-10-21T05:30:37.714 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40785.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.40785.log.gz /var/log/ceph/ceph-client.admin.40432.log 2023-10-21T05:30:37.714 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-10-21T05:30:37.714 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49841.log 2023-10-21T05:30:37.714 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46552.log.gz 2023-10-21T05:30:37.715 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38547.log 2023-10-21T05:30:37.715 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40432.log.gz 2023-10-21T05:30:37.715 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50272.log 2023-10-21T05:30:37.715 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49841.log.gz 2023-10-21T05:30:37.716 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38547.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.38547.log.gz -- 2023-10-21T05:30:37.716 INFO:teuthology.orchestra.run.smithi133.stderr: /var/log/ceph/ceph-client.admin.42184.log 2023-10-21T05:30:37.716 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42083.log 2023-10-21T05:30:37.716 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50272.log.gz 2023-10-21T05:30:37.717 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38507.log 2023-10-21T05:30:37.717 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42184.log.gz 2023-10-21T05:30:37.717 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41900.log 2023-10-21T05:30:37.717 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42083.log.gz 2023-10-21T05:30:37.718 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51591.log 2023-10-21T05:30:37.718 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38507.log.gz 2023-10-21T05:30:37.718 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41900.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.41900.log.gz -- 2023-10-21T05:30:37.719 INFO:teuthology.orchestra.run.smithi133.stderr: /var/log/ceph/ceph-client.admin.52030.log 2023-10-21T05:30:37.719 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44605.log 2023-10-21T05:30:37.719 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51591.log.gz 2023-10-21T05:30:37.719 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49083.log 2023-10-21T05:30:37.720 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52030.log.gz 2023-10-21T05:30:37.720 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50785.log 2023-10-21T05:30:37.720 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44605.log.gz 2023-10-21T05:30:37.720 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37712.log 2023-10-21T05:30:37.721 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49083.log.gz 2023-10-21T05:30:37.721 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50785.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.49599.log -- replaced with /var/log/ceph/ceph-client.admin.50785.log.gz 2023-10-21T05:30:37.721 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-10-21T05:30:37.721 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37712.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37712.log.gz 2023-10-21T05:30:37.721 INFO:teuthology.orchestra.run.smithi133.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52321.log 2023-10-21T05:30:37.722 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52108.log 2023-10-21T05:30:37.722 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49599.log.gz 2023-10-21T05:30:37.722 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45735.log 2023-10-21T05:30:37.723 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52321.log.gz 2023-10-21T05:30:37.723 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52108.log.gz 2023-10-21T05:30:37.723 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50664.log 2023-10-21T05:30:37.723 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43875.log 2023-10-21T05:30:37.724 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45735.log.gz 2023-10-21T05:30:37.724 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54191.log 2023-10-21T05:30:37.724 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50664.log.gz 2023-10-21T05:30:37.725 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41809.log 2023-10-21T05:30:37.725 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43875.log.gz 2023-10-21T05:30:37.725 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39158.log 2023-10-21T05:30:37.725 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54191.log.gz 2023-10-21T05:30:37.726 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41809.log.gz 2023-10-21T05:30:37.726 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51271.log 2023-10-21T05:30:37.726 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39158.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-21T05:30:37.727 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39158.log.gz 2023-10-21T05:30:37.727 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44788.log 2023-10-21T05:30:37.727 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51271.log.gz 2023-10-21T05:30:37.727 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52877.log 2023-10-21T05:30:37.728 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49750.log 2023-10-21T05:30:37.728 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44788.log.gz 2023-10-21T05:30:37.728 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46829.log 2023-10-21T05:30:37.728 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52877.log.gz 2023-10-21T05:30:37.729 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50482.log 2023-10-21T05:30:37.729 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49750.log.gz 2023-10-21T05:30:37.729 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48076.log 2023-10-21T05:30:37.729 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46829.log.gz 2023-10-21T05:30:37.730 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50482.log.gz 2023-10-21T05:30:37.730 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50937.log 2023-10-21T05:30:37.730 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51952.log 2023-10-21T05:30:37.730 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48076.log.gz 2023-10-21T05:30:37.731 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51385.log 2023-10-21T05:30:37.731 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50937.log.gz 2023-10-21T05:30:37.731 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51952.log.gz 2023-10-21T05:30:37.732 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49389.log 2023-10-21T05:30:37.732 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46036.log 2023-10-21T05:30:37.732 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51385.log.gz 2023-10-21T05:30:37.732 INFO:teuthology.orchestra.run.smithi133.stderr:gzip/var/log/ceph/ceph-client.admin.49389.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38638.log 2023-10-21T05:30:37.733 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49389.log.gz 2023-10-21T05:30:37.733 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39374.log 2023-10-21T05:30:37.733 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46036.log.gz 2023-10-21T05:30:37.733 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42747.log 2023-10-21T05:30:37.734 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38638.log.gz 2023-10-21T05:30:37.734 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39394.log 2023-10-21T05:30:37.734 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39374.log.gz 2023-10-21T05:30:37.735 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42747.log.gz 2023-10-21T05:30:37.735 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47621.log 2023-10-21T05:30:37.735 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54272.log 2023-10-21T05:30:37.735 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39394.log.gz 2023-10-21T05:30:37.736 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39519.log 2023-10-21T05:30:37.736 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47621.log.gz 2023-10-21T05:30:37.736 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54272.log.gz 2023-10-21T05:30:37.736 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51060.log 2023-10-21T05:30:37.737 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41032.log 2023-10-21T05:30:37.737 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39519.log.gz 2023-10-21T05:30:37.737 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49996.log 2023-10-21T05:30:37.737 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51060.log.gz 2023-10-21T05:30:37.738 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39357.log 2023-10-21T05:30:37.738 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41032.log.gz 2023-10-21T05:30:37.738 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49419.log 2023-10-21T05:30:37.739 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49996.log.gz 2023-10-21T05:30:37.739 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40541.log 2023-10-21T05:30:37.739 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39357.log.gz 2023-10-21T05:30:37.739 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39266.log 2023-10-21T05:30:37.740 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49419.log.gz 2023-10-21T05:30:37.740 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47893.log 2023-10-21T05:30:37.740 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40541.log.gz 2023-10-21T05:30:37.740 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50211.log 2023-10-21T05:30:37.741 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39266.log.gz 2023-10-21T05:30:37.741 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41727.log 2023-10-21T05:30:37.741 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47893.log.gz 2023-10-21T05:30:37.741 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50211.log.gz 2023-10-21T05:30:37.742 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38564.log 2023-10-21T05:30:37.742 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46708.log 2023-10-21T05:30:37.742 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41727.log.gz 2023-10-21T05:30:37.743 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41001.log 2023-10-21T05:30:37.743 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38564.log.gz 2023-10-21T05:30:37.743 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39778.log 2023-10-21T05:30:37.743 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46708.log.gz 2023-10-21T05:30:37.744 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47712.log 2023-10-21T05:30:37.744 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41001.log.gz 2023-10-21T05:30:37.744 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39778.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.39778.log.gz /var/log/ceph/ceph-client.admin.45915.log 2023-10-21T05:30:37.745 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-10-21T05:30:37.745 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45945.log 2023-10-21T05:30:37.745 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47712.log.gz 2023-10-21T05:30:37.745 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45915.log.gz 2023-10-21T05:30:37.746 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40116.log 2023-10-21T05:30:37.746 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45945.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-21T05:30:37.746 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45945.log.gz 2023-10-21T05:30:37.747 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37634.log 2023-10-21T05:30:37.747 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40116.log.gz 2023-10-21T05:30:37.747 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46920.log 2023-10-21T05:30:37.747 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44363.log 2023-10-21T05:30:37.748 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37634.log.gz 2023-10-21T05:30:37.748 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45337.log 2023-10-21T05:30:37.748 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46920.log.gz 2023-10-21T05:30:37.748 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39502.log 2023-10-21T05:30:37.749 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44363.log.gz 2023-10-21T05:30:37.749 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48228.log 2023-10-21T05:30:37.749 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45337.log.gz 2023-10-21T05:30:37.749 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46249.log 2023-10-21T05:30:37.750 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39502.log.gz 2023-10-21T05:30:37.750 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52453.log 2023-10-21T05:30:37.750 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48228.log.gz 2023-10-21T05:30:37.751 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37410.log 2023-10-21T05:30:37.751 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46249.log.gz 2023-10-21T05:30:37.751 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53869.log 2023-10-21T05:30:37.751 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52453.log.gz 2023-10-21T05:30:37.752 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53199.log 2023-10-21T05:30:37.752 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37410.log.gz 2023-10-21T05:30:37.752 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53839.log 2023-10-21T05:30:37.752 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53869.log.gz 2023-10-21T05:30:37.753 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48261.log 2023-10-21T05:30:37.753 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53199.log.gz 2023-10-21T05:30:37.753 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53171.log 2023-10-21T05:30:37.754 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53839.log.gz 2023-10-21T05:30:37.754 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46860.log 2023-10-21T05:30:37.754 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48261.log.gz 2023-10-21T05:30:37.754 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42329.log 2023-10-21T05:30:37.755 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53171.log.gz 2023-10-21T05:30:37.755 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48961.log 2023-10-21T05:30:37.755 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46860.log.gz 2023-10-21T05:30:37.756 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44851.log 2023-10-21T05:30:37.756 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42329.log.gz 2023-10-21T05:30:37.756 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2023-10-21T05:30:37.756 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48961.log.gz 2023-10-21T05:30:37.757 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49569.log 2023-10-21T05:30:37.757 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44851.log.gz 2023-10-21T05:30:37.757 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40920.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz /var/log/ceph/ceph-client.admin.47344.log 2023-10-21T05:30:37.757 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-10-21T05:30:37.758 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40601.log 2023-10-21T05:30:37.758 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49569.log.gz 2023-10-21T05:30:37.758 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50512.log 2023-10-21T05:30:37.759 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47344.log.gz 2023-10-21T05:30:37.759 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44697.log 2023-10-21T05:30:37.759 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40601.log.gz 2023-10-21T05:30:37.759 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47681.log 2023-10-21T05:30:37.760 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50512.log.gz 2023-10-21T05:30:37.870 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43656.log 2023-10-21T05:30:37.870 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44697.log.gz 2023-10-21T05:30:37.871 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51495.log 2023-10-21T05:30:37.871 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47681.log.gz 2023-10-21T05:30:37.871 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48806.log 2023-10-21T05:30:37.871 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43656.log.gz 2023-10-21T05:30:37.872 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34436.log 2023-10-21T05:30:37.872 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51495.log.gz 2023-10-21T05:30:37.872 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52799.log 2023-10-21T05:30:37.873 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48806.log.gz 2023-10-21T05:30:37.873 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.34436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39735.log 2023-10-21T05:30:37.873 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34436.log.gz 2023-10-21T05:30:37.873 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40571.log 2023-10-21T05:30:37.874 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52799.log.gz 2023-10-21T05:30:37.874 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37632.log 2023-10-21T05:30:37.874 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39735.log.gz 2023-10-21T05:30:37.875 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52559.log 2023-10-21T05:30:37.875 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40571.log.gz 2023-10-21T05:30:37.875 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.37632.log: -- /var/log/ceph/ceph-client.admin.52987.log 2023-10-21T05:30:37.875 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37632.log.gz 2023-10-21T05:30:37.876 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37711.log 2023-10-21T05:30:37.876 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52559.log.gz 2023-10-21T05:30:37.876 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52987.log.gzgzip 2023-10-21T05:30:37.876 INFO:teuthology.orchestra.run.smithi133.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49206.log 2023-10-21T05:30:37.877 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50392.log 2023-10-21T05:30:37.877 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37711.log.gz 2023-10-21T05:30:37.877 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39686.log 2023-10-21T05:30:37.878 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49206.log.gz 2023-10-21T05:30:37.878 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50422.log 2023-10-21T05:30:37.878 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50392.log.gz 2023-10-21T05:30:37.878 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41563.log 2023-10-21T05:30:37.879 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39686.log.gz 2023-10-21T05:30:37.879 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43164.log 2023-10-21T05:30:37.879 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50422.log.gz 2023-10-21T05:30:37.880 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41563.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41563.log.gz 2023-10-21T05:30:37.880 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48167.log 2023-10-21T05:30:37.880 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46005.log 2023-10-21T05:30:37.880 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43164.log.gz 2023-10-21T05:30:37.881 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48167.log.gz 2023-10-21T05:30:37.881 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49902.log 2023-10-21T05:30:37.881 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45825.log 2023-10-21T05:30:37.882 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46005.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46005.log.gz 2023-10-21T05:30:37.882 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49902.log.gz 2023-10-21T05:30:37.882 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53652.log 2023-10-21T05:30:37.882 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52507.log 2023-10-21T05:30:37.883 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45825.log.gz 2023-10-21T05:30:37.883 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53652.log.gz 2023-10-21T05:30:37.883 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50815.log 2023-10-21T05:30:37.884 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51513.log 2023-10-21T05:30:37.884 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52507.log.gz 2023-10-21T05:30:37.884 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42270.log 2023-10-21T05:30:37.884 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50815.log.gz 2023-10-21T05:30:37.885 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46799.log 2023-10-21T05:30:37.885 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51513.log.gz 2023-10-21T05:30:37.885 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49358.log 2023-10-21T05:30:37.885 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42270.log.gz 2023-10-21T05:30:37.886 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39104.log 2023-10-21T05:30:37.886 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46799.log.gz 2023-10-21T05:30:37.886 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49358.log.gz 2023-10-21T05:30:37.887 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41972.log 2023-10-21T05:30:37.887 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42297.log 2023-10-21T05:30:37.887 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39104.log.gz 2023-10-21T05:30:37.887 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47010.log 2023-10-21T05:30:37.888 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41972.log.gz 2023-10-21T05:30:37.888 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46617.log 2023-10-21T05:30:37.888 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42297.log.gz 2023-10-21T05:30:37.888 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50604.log 2023-10-21T05:30:37.889 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47010.log.gz 2023-10-21T05:30:37.889 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42246.log 2023-10-21T05:30:37.889 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46617.log.gz 2023-10-21T05:30:37.889 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46279.log 2023-10-21T05:30:37.890 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50604.log.gz 2023-10-21T05:30:37.890 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37471.log 2023-10-21T05:30:37.890 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42246.log.gz 2023-10-21T05:30:37.891 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45095.log 2023-10-21T05:30:37.891 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46279.log.gz 2023-10-21T05:30:37.891 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52851.log 2023-10-21T05:30:37.891 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37471.log.gz 2023-10-21T05:30:37.892 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44515.log 2023-10-21T05:30:37.892 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45095.log.gz 2023-10-21T05:30:37.892 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47283.log 2023-10-21T05:30:37.893 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52851.log.gz 2023-10-21T05:30:37.893 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45398.log 2023-10-21T05:30:37.893 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44515.log.gz 2023-10-21T05:30:37.893 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47803.log 2023-10-21T05:30:37.894 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47283.log.gz 2023-10-21T05:30:37.894 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53785.log 2023-10-21T05:30:37.894 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45398.log.gz 2023-10-21T05:30:37.894 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46584.log 2023-10-21T05:30:37.895 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47803.log.gz 2023-10-21T05:30:37.895 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44393.log 2023-10-21T05:30:37.895 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53785.log.gz 2023-10-21T05:30:37.895 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52744.log 2023-10-21T05:30:37.896 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46584.log.gz 2023-10-21T05:30:37.896 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44393.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44393.log.gz 2023-10-21T05:30:37.896 INFO:teuthology.orchestra.run.smithi133.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51874.log 2023-10-21T05:30:37.897 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43752.log 2023-10-21T05:30:37.897 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52744.log.gz 2023-10-21T05:30:37.897 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49719.log 2023-10-21T05:30:37.897 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51874.log.gz 2023-10-21T05:30:37.898 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46522.log 2023-10-21T05:30:37.898 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43752.log.gz 2023-10-21T05:30:37.898 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49719.log.gz 2023-10-21T05:30:37.899 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44332.log 2023-10-21T05:30:37.899 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43535.log 2023-10-21T05:30:37.899 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46522.log.gz 2023-10-21T05:30:37.899 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44332.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.44332.log.gz 2023-10-21T05:30:37.900 INFO:teuthology.orchestra.run.smithi133.stderr: /var/log/ceph/ceph-client.admin.52773.log 2023-10-21T05:30:37.900 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53013.log 2023-10-21T05:30:37.900 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43535.log.gz 2023-10-21T05:30:37.900 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39033.log 2023-10-21T05:30:37.901 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52773.log.gz 2023-10-21T05:30:37.901 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53979.log 2023-10-21T05:30:37.901 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53013.log.gz 2023-10-21T05:30:37.901 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51926.log 2023-10-21T05:30:37.902 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39033.log.gz 2023-10-21T05:30:37.902 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38979.log 2023-10-21T05:30:37.902 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53979.log.gz 2023-10-21T05:30:37.903 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53334.log 2023-10-21T05:30:37.903 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51926.log.gz 2023-10-21T05:30:37.903 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43949.log 2023-10-21T05:30:37.903 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38979.log.gz 2023-10-21T05:30:37.904 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52479.log 2023-10-21T05:30:37.904 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53334.log.gz 2023-10-21T05:30:37.904 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45065.log 2023-10-21T05:30:37.904 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43949.log.gz 2023-10-21T05:30:37.905 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39286.log 2023-10-21T05:30:37.905 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52479.log.gz 2023-10-21T05:30:37.905 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41779.log 2023-10-21T05:30:37.906 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45065.log.gz 2023-10-21T05:30:37.906 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38942.log 2023-10-21T05:30:37.906 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39286.log.gz 2023-10-21T05:30:37.906 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39303.log 2023-10-21T05:30:37.907 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41779.log.gz 2023-10-21T05:30:37.907 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53492.log 2023-10-21T05:30:37.907 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38942.log.gz 2023-10-21T05:30:37.907 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43456.log 2023-10-21T05:30:37.908 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39303.log.gz 2023-10-21T05:30:37.908 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43565.log 2023-10-21T05:30:37.908 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53492.log.gz 2023-10-21T05:30:37.908 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51737.log 2023-10-21T05:30:37.909 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43456.log.gz 2023-10-21T05:30:37.909 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39178.log 2023-10-21T05:30:37.909 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43565.log.gz 2023-10-21T05:30:37.910 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37574.log 2023-10-21T05:30:37.910 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51737.log.gz 2023-10-21T05:30:37.910 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38871.log 2023-10-21T05:30:37.910 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39178.log.gz 2023-10-21T05:30:37.911 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45458.log 2023-10-21T05:30:37.911 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37574.log.gz 2023-10-21T05:30:37.911 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37963.log 2023-10-21T05:30:37.911 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38871.log.gz 2023-10-21T05:30:37.912 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43626.log 2023-10-21T05:30:37.912 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45458.log.gz 2023-10-21T05:30:37.912 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44245.log 2023-10-21T05:30:37.912 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37963.log.gz 2023-10-21T05:30:37.913 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46127.log 2023-10-21T05:30:37.913 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43626.log.gz 2023-10-21T05:30:37.913 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44245.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.51719.log -- replaced with /var/log/ceph/ceph-client.admin.44245.log.gz 2023-10-21T05:30:37.914 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-10-21T05:30:37.914 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37548.log 2023-10-21T05:30:37.914 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46127.log.gz 2023-10-21T05:30:37.914 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44637.log 2023-10-21T05:30:37.915 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51719.log.gz 2023-10-21T05:30:37.915 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54006.log 2023-10-21T05:30:37.915 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37548.log.gz 2023-10-21T05:30:37.917 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39016.log 2023-10-21T05:30:37.917 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44637.log.gz 2023-10-21T05:30:37.917 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47923.log 2023-10-21T05:30:37.918 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54006.log.gz 2023-10-21T05:30:37.918 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39016.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.39016.log.gz -- 2023-10-21T05:30:37.918 INFO:teuthology.orchestra.run.smithi133.stderr: /var/log/ceph/ceph-client.admin.37704.log 2023-10-21T05:30:37.918 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52961.log 2023-10-21T05:30:37.919 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47923.log.gz 2023-10-21T05:30:37.919 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49689.log 2023-10-21T05:30:37.919 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37704.log.gz 2023-10-21T05:30:37.920 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53759.log 2023-10-21T05:30:37.920 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52961.log.gz 2023-10-21T05:30:37.920 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47252.log 2023-10-21T05:30:37.920 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49689.log.gz 2023-10-21T05:30:37.921 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53759.log.gz 2023-10-21T05:30:37.921 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38527.log 2023-10-21T05:30:37.921 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49328.log 2023-10-21T05:30:37.921 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47252.log.gz 2023-10-21T05:30:37.922 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38908.log 2023-10-21T05:30:37.922 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38527.log.gz 2023-10-21T05:30:37.922 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49328.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.49328.log.gz 2023-10-21T05:30:37.922 INFO:teuthology.orchestra.run.smithi133.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54113.log 2023-10-21T05:30:37.923 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40334.log 2023-10-21T05:30:37.923 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38908.log.gz 2023-10-21T05:30:37.923 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39948.log 2023-10-21T05:30:37.924 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54113.log.gz 2023-10-21T05:30:37.924 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40334.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40334.log.gz -- 2023-10-21T05:30:37.924 INFO:teuthology.orchestra.run.smithi133.stderr: /var/log/ceph/ceph-client.admin.53572.log 2023-10-21T05:30:37.924 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39948.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-21T05:30:37.925 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39948.log.gz 2023-10-21T05:30:37.925 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48656.log 2023-10-21T05:30:37.925 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53572.log.gz 2023-10-21T05:30:37.925 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39482.log 2023-10-21T05:30:37.926 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51421.log 2023-10-21T05:30:37.926 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48656.log.gz 2023-10-21T05:30:37.926 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54058.log 2023-10-21T05:30:37.927 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39482.log.gz 2023-10-21T05:30:37.927 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51181.log 2023-10-21T05:30:37.927 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51421.log.gz 2023-10-21T05:30:37.927 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49175.log 2023-10-21T05:30:37.928 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54058.log.gz 2023-10-21T05:30:37.928 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39906.log 2023-10-21T05:30:37.928 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51181.log.gz 2023-10-21T05:30:37.928 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51822.log 2023-10-21T05:30:37.929 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49175.log.gz 2023-10-21T05:30:37.929 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37633.log 2023-10-21T05:30:37.929 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39906.log.gz 2023-10-21T05:30:37.929 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37782.log 2023-10-21T05:30:37.930 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51822.log.gz 2023-10-21T05:30:37.930 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41479.log 2023-10-21T05:30:37.930 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37633.log.gz 2023-10-21T05:30:37.931 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54428.log 2023-10-21T05:30:37.931 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37782.log.gz 2023-10-21T05:30:37.931 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41154.log 2023-10-21T05:30:37.931 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41479.log.gz 2023-10-21T05:30:37.932 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39050.log 2023-10-21T05:30:37.932 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54428.log.gz 2023-10-21T05:30:37.932 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49629.log 2023-10-21T05:30:37.932 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41154.log.gz 2023-10-21T05:30:37.933 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51403.log 2023-10-21T05:30:37.933 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39050.log.gz 2023-10-21T05:30:37.933 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49629.log.gz 2023-10-21T05:30:37.933 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41124.log 2023-10-21T05:30:37.934 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53546.log 2023-10-21T05:30:37.934 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51403.log.gz 2023-10-21T05:30:37.934 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45155.log 2023-10-21T05:30:37.935 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41124.log.gz 2023-10-21T05:30:37.935 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37496.log 2023-10-21T05:30:37.935 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53546.log.gz 2023-10-21T05:30:37.935 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46097.log 2023-10-21T05:30:37.936 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45155.log.gz 2023-10-21T05:30:37.936 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52585.log 2023-10-21T05:30:37.936 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37496.log.gz 2023-10-21T05:30:37.936 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40032.log 2023-10-21T05:30:37.937 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46097.log.gz 2023-10-21T05:30:37.937 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37912.log 2023-10-21T05:30:37.937 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52585.log.gz 2023-10-21T05:30:37.937 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47101.log 2023-10-21T05:30:37.938 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40032.log.gz 2023-10-21T05:30:37.938 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41229.log 2023-10-21T05:30:37.938 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37912.log.gz 2023-10-21T05:30:37.939 INFO:teuthology.orchestra.run.smithi133.stderr:gzip/var/log/ceph/ceph-client.admin.47101.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41317.log 2023-10-21T05:30:37.939 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47101.log.gz 2023-10-21T05:30:37.939 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38817.log 2023-10-21T05:30:37.939 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41229.log.gz 2023-10-21T05:30:37.940 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43260.log 2023-10-21T05:30:37.940 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41317.log.gz 2023-10-21T05:30:37.940 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45216.log 2023-10-21T05:30:37.940 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38817.log.gz 2023-10-21T05:30:37.941 INFO:teuthology.orchestra.run.smithi133.stderr:gzip/var/log/ceph/ceph-client.admin.43260.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48566.log 2023-10-21T05:30:37.941 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43260.log.gz 2023-10-21T05:30:37.941 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44193.log 2023-10-21T05:30:37.941 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45216.log.gz 2023-10-21T05:30:37.942 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41839.log 2023-10-21T05:30:37.942 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48566.log.gz 2023-10-21T05:30:37.942 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52082.log 2023-10-21T05:30:37.943 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44193.log.gz 2023-10-21T05:30:37.943 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49659.log 2023-10-21T05:30:37.943 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41839.log.gz 2023-10-21T05:30:37.943 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38925.log 2023-10-21T05:30:37.944 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52082.log.gz 2023-10-21T05:30:37.944 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39141.log 2023-10-21T05:30:37.944 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49659.log.gz 2023-10-21T05:30:37.945 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51535.log 2023-10-21T05:30:37.945 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38925.log.gz 2023-10-21T05:30:37.945 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43905.log 2023-10-21T05:30:37.945 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39141.log.gz 2023-10-21T05:30:37.946 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52217.log 2023-10-21T05:30:37.946 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51535.log.gz 2023-10-21T05:30:37.946 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45551.log 2023-10-21T05:30:37.946 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43905.log.gz 2023-10-21T05:30:37.947 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53119.log 2023-10-21T05:30:37.947 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52217.log.gz 2023-10-21T05:30:37.947 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47162.log 2023-10-21T05:30:37.948 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45551.log.gz 2023-10-21T05:30:37.948 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48198.log 2023-10-21T05:30:37.948 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53119.log.gz 2023-10-21T05:30:37.948 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37786.log 2023-10-21T05:30:37.949 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47162.log.gz 2023-10-21T05:30:37.949 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47192.log 2023-10-21T05:30:37.949 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48198.log.gz 2023-10-21T05:30:37.949 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41625.log 2023-10-21T05:30:37.950 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37786.log.gz 2023-10-21T05:30:37.950 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38996.log 2023-10-21T05:30:37.950 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47192.log.gz 2023-10-21T05:30:37.951 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53923.log 2023-10-21T05:30:37.951 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41625.log.gz 2023-10-21T05:30:37.951 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40631.log 2023-10-21T05:30:37.951 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38996.log.gz 2023-10-21T05:30:37.952 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53923.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-21T05:30:37.952 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53923.log.gz 2023-10-21T05:30:37.952 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38473.log 2023-10-21T05:30:37.952 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40631.log.gz 2023-10-21T05:30:37.953 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53360.log 2023-10-21T05:30:37.953 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52825.log 2023-10-21T05:30:37.953 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38473.log.gz 2023-10-21T05:30:37.954 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52637.log 2023-10-21T05:30:37.954 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53360.log.gz 2023-10-21T05:30:37.954 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51302.log 2023-10-21T05:30:37.954 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52825.log.gz 2023-10-21T05:30:37.955 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44119.log 2023-10-21T05:30:37.955 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52637.log.gz 2023-10-21T05:30:37.955 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48505.log 2023-10-21T05:30:37.955 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51302.log.gz 2023-10-21T05:30:37.956 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45642.log 2023-10-21T05:30:37.956 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44119.log.gz 2023-10-21T05:30:37.956 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44424.log 2023-10-21T05:30:37.957 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48505.log.gz 2023-10-21T05:30:37.957 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45642.log.gz 2023-10-21T05:30:37.957 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43595.log 2023-10-21T05:30:37.957 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42771.log 2023-10-21T05:30:37.958 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44424.log.gz 2023-10-21T05:30:37.958 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51627.log 2023-10-21T05:30:37.958 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43595.log.gz 2023-10-21T05:30:37.958 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42771.log.gz 2023-10-21T05:30:37.959 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42390.log 2023-10-21T05:30:37.959 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47591.log 2023-10-21T05:30:37.959 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51627.log.gz 2023-10-21T05:30:37.959 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42390.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-21T05:30:37.960 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42390.log.gz 2023-10-21T05:30:37.960 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38584.log 2023-10-21T05:30:37.960 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47591.log.gz 2023-10-21T05:30:37.960 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51121.log 2023-10-21T05:30:37.961 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49144.log 2023-10-21T05:30:37.961 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38584.log.gz 2023-10-21T05:30:37.961 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39212.log 2023-10-21T05:30:37.961 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40484.log 2023-10-21T05:30:37.962 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49144.log.gz 2023-10-21T05:30:37.962 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39212.log.gzgzip 2023-10-21T05:30:37.962 INFO:teuthology.orchestra.run.smithi133.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50724.log 2023-10-21T05:30:37.963 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39448.log 2023-10-21T05:30:37.963 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40484.log.gz 2023-10-21T05:30:37.963 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47953.log 2023-10-21T05:30:37.963 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50724.log.gz 2023-10-21T05:30:37.964 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41451.log 2023-10-21T05:30:37.964 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39448.log.gz 2023-10-21T05:30:37.964 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42462.log 2023-10-21T05:30:37.964 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47953.log.gz 2023-10-21T05:30:37.965 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40158.log 2023-10-21T05:30:37.965 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41451.log.gz 2023-10-21T05:30:37.965 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48046.log 2023-10-21T05:30:37.965 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42462.log.gz 2023-10-21T05:30:37.966 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47467.log 2023-10-21T05:30:37.966 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40158.log.gz 2023-10-21T05:30:37.966 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48046.log.gz 2023-10-21T05:30:37.967 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52718.log 2023-10-21T05:30:37.967 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40848.log 2023-10-21T05:30:37.967 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47467.log.gz 2023-10-21T05:30:37.967 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53228.log 2023-10-21T05:30:37.968 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52718.log.gz 2023-10-21T05:30:37.968 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40848.log.gz 2023-10-21T05:30:37.968 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41347.log 2023-10-21T05:30:37.968 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44912.log 2023-10-21T05:30:37.969 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53228.log.gz 2023-10-21T05:30:37.969 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37860.log 2023-10-21T05:30:37.969 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41347.log.gz 2023-10-21T05:30:37.969 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44912.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.44912.log.gz --verbose 2023-10-21T05:30:37.970 INFO:teuthology.orchestra.run.smithi133.stderr: -- /var/log/ceph/ceph-client.admin.45276.log 2023-10-21T05:30:37.970 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44821.log 2023-10-21T05:30:37.970 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37860.log.gz 2023-10-21T05:30:37.970 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41503.log 2023-10-21T05:30:37.971 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45276.log.gz 2023-10-21T05:30:37.971 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54165.log 2023-10-21T05:30:37.971 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44821.log.gz 2023-10-21T05:30:37.971 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41752.log 2023-10-21T05:30:37.972 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41503.log.gz 2023-10-21T05:30:37.972 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47314.log 2023-10-21T05:30:37.972 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54165.log.gz 2023-10-21T05:30:37.973 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48776.log 2023-10-21T05:30:37.973 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41752.log.gz 2023-10-21T05:30:37.973 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52243.log 2023-10-21T05:30:37.973 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47314.log.gz 2023-10-21T05:30:37.974 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34578.log 2023-10-21T05:30:37.974 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48776.log.gz 2023-10-21T05:30:37.974 INFO:teuthology.orchestra.run.smithi133.stderr: 27.2% -- replaced with /var/log/ceph/ceph-client.admin.51121.log.gz 2023-10-21T05:30:37.974 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40950.log 2023-10-21T05:30:37.975 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52243.log.gz 2023-10-21T05:30:37.975 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.34578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49114.log 2023-10-21T05:30:37.975 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34578.log.gz 2023-10-21T05:30:37.975 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40950.log.gz 2023-10-21T05:30:37.976 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52663.log 2023-10-21T05:30:37.976 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45765.log 2023-10-21T05:30:37.976 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49114.log.gz 2023-10-21T05:30:37.976 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46678.log 2023-10-21T05:30:37.977 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52663.log.gz 2023-10-21T05:30:37.977 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44945.log 2023-10-21T05:30:37.977 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45765.log.gz 2023-10-21T05:30:37.978 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48626.log 2023-10-21T05:30:37.978 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46678.log.gz 2023-10-21T05:30:37.978 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49780.log 2023-10-21T05:30:37.978 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44945.log.gz 2023-10-21T05:30:37.979 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53732.log 2023-10-21T05:30:37.979 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48626.log.gz 2023-10-21T05:30:37.979 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50362.log 2023-10-21T05:30:37.979 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49780.log.gz 2023-10-21T05:30:37.980 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54324.log 2023-10-21T05:30:37.980 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53732.log.gz 2023-10-21T05:30:37.980 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47833.log 2023-10-21T05:30:37.981 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50362.log.gz 2023-10-21T05:30:37.981 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38780.log 2023-10-21T05:30:37.981 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54324.log.gz 2023-10-21T05:30:37.981 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46067.log 2023-10-21T05:30:37.982 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47833.log.gz 2023-10-21T05:30:37.982 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53520.log 2023-10-21T05:30:37.982 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38780.log.gz 2023-10-21T05:30:37.982 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51029.log 2023-10-21T05:30:37.983 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46067.log.gz 2023-10-21T05:30:37.983 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51091.log 2023-10-21T05:30:37.983 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53520.log.gz 2023-10-21T05:30:37.984 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34530.log 2023-10-21T05:30:37.984 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51029.log.gz 2023-10-21T05:30:37.984 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44149.log 2023-10-21T05:30:37.984 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51091.log.gz 2023-10-21T05:30:37.985 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.34530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38692.log 2023-10-21T05:30:37.985 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34530.log.gz 2023-10-21T05:30:37.985 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38888.log 2023-10-21T05:30:37.986 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44149.log.gz 2023-10-21T05:30:37.986 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49268.log 2023-10-21T05:30:37.986 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38692.log.gz 2023-10-21T05:30:37.986 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47559.log 2023-10-21T05:30:37.987 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38888.log.gz 2023-10-21T05:30:37.987 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43484.log 2023-10-21T05:30:37.987 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49268.log.gz 2023-10-21T05:30:37.987 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44272.log 2023-10-21T05:30:37.988 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47559.log.gz 2023-10-21T05:30:37.988 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43484.log.gz 2023-10-21T05:30:37.988 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49539.log 2023-10-21T05:30:37.989 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40376.log 2023-10-21T05:30:37.989 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44272.log.gz 2023-10-21T05:30:37.989 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51571.log 2023-10-21T05:30:37.989 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49539.log.gz 2023-10-21T05:30:37.990 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39428.log 2023-10-21T05:30:37.990 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40376.log.gz 2023-10-21T05:30:37.990 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51571.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.48292.log 2023-10-21T05:30:37.990 INFO:teuthology.orchestra.run.smithi133.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51571.log.gz 2023-10-21T05:30:37.991 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54217.log 2023-10-21T05:30:37.991 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39428.log.gz 2023-10-21T05:30:37.991 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48323.log 2023-10-21T05:30:37.992 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48292.log.gz 2023-10-21T05:30:37.992 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51477.log 0.0% 2023-10-21T05:30:37.992 INFO:teuthology.orchestra.run.smithi133.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54217.log.gz 2023-10-21T05:30:37.992 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39574.log 2023-10-21T05:30:37.993 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48323.log.gz 2023-10-21T05:30:37.993 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38763.log 2023-10-21T05:30:37.993 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51477.log.gz 2023-10-21T05:30:37.994 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51681.log 2023-10-21T05:30:37.994 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39574.log.gz 2023-10-21T05:30:37.994 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42513.log 2023-10-21T05:30:37.994 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38763.log.gz 2023-10-21T05:30:37.995 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52691.log 2023-10-21T05:30:37.995 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51681.log.gz 2023-10-21T05:30:37.995 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40676.log 2023-10-21T05:30:37.995 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42513.log.gz 2023-10-21T05:30:37.996 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50332.log 2023-10-21T05:30:37.996 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52691.log.gz 2023-10-21T05:30:37.996 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45246.log 2023-10-21T05:30:37.997 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40676.log.gz 2023-10-21T05:30:37.997 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41996.log 2023-10-21T05:30:37.997 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50332.log.gz 2023-10-21T05:30:37.997 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41944.log 2023-10-21T05:30:37.998 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45246.log.gz 2023-10-21T05:30:37.998 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52904.log 2023-10-21T05:30:37.998 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41996.log.gz 2023-10-21T05:30:37.999 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48415.log 2023-10-21T05:30:37.999 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41944.log.gz 2023-10-21T05:30:37.999 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53438.log 2023-10-21T05:30:37.999 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52904.log.gz 2023-10-21T05:30:38.000 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45307.log 2023-10-21T05:30:38.000 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48415.log.gz 2023-10-21T05:30:38.000 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39232.log 2023-10-21T05:30:38.000 INFO:teuthology.orchestra.run.smithi133.stderr: 26.6% -- replaced with /var/log/ceph/ceph-client.admin.53438.log.gz 2023-10-21T05:30:38.001 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45520.log 2023-10-21T05:30:38.001 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45307.log.gz 2023-10-21T05:30:38.001 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51699.log 2023-10-21T05:30:38.002 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39232.log.gz 2023-10-21T05:30:38.002 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45520.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45520.log.gz -5 2023-10-21T05:30:38.002 INFO:teuthology.orchestra.run.smithi133.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53281.log 2023-10-21T05:30:38.002 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46400.log 2023-10-21T05:30:38.003 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51699.log.gz 2023-10-21T05:30:38.003 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40818.log 2023-10-21T05:30:38.003 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53281.log.gz 2023-10-21T05:30:38.004 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50302.log 2023-10-21T05:30:38.004 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46400.log.gz 2023-10-21T05:30:38.004 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53465.log 2023-10-21T05:30:38.004 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40818.log.gz 2023-10-21T05:30:38.005 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50302.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50302.log.gz -5 2023-10-21T05:30:38.005 INFO:teuthology.orchestra.run.smithi133.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50755.log 2023-10-21T05:30:38.005 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53813.log 2023-10-21T05:30:38.005 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53465.log.gz 2023-10-21T05:30:38.006 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50755.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-21T05:30:38.006 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50755.log.gz 2023-10-21T05:30:38.006 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40511.log 2023-10-21T05:30:38.007 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53813.log.gz 2023-10-21T05:30:38.007 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45855.log 2023-10-21T05:30:38.007 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53067.log 2023-10-21T05:30:38.007 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40511.log.gz 2023-10-21T05:30:38.008 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50906.log 2023-10-21T05:30:38.008 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45855.log.gz 2023-10-21T05:30:38.008 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53680.log 2023-10-21T05:30:38.009 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53067.log.gz 2023-10-21T05:30:38.009 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50906.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30878.log 2023-10-21T05:30:38.009 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50906.log.gz 2023-10-21T05:30:38.009 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44302.log 2023-10-21T05:30:38.010 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53680.log.gz 2023-10-21T05:30:38.010 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph.tmp-client.admin.30878.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.48716.log 2023-10-21T05:30:38.010 INFO:teuthology.orchestra.run.smithi133.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.30878.log.gz 2023-10-21T05:30:38.010 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54084.log 2023-10-21T05:30:38.011 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44302.log.gz 2023-10-21T05:30:38.011 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53254.log 2023-10-21T05:30:38.011 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48716.log.gz 2023-10-21T05:30:38.012 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48384.log 2023-10-21T05:30:38.012 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54084.log.gz 2023-10-21T05:30:38.012 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54139.log 2023-10-21T05:30:38.012 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53254.log.gz 2023-10-21T05:30:38.013 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49935.log 2023-10-21T05:30:38.013 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48384.log.gz 2023-10-21T05:30:38.013 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42023.log 2023-10-21T05:30:38.013 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54139.log.gz 2023-10-21T05:30:38.014 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.49935.log: -- /var/log/ceph/ceph-client.admin.50998.log 2023-10-21T05:30:38.014 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49935.log.gz 2023-10-21T05:30:38.014 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40755.log 2023-10-21T05:30:38.015 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42023.log.gz 2023-10-21T05:30:38.015 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41377.log 2023-10-21T05:30:38.015 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50998.log.gz 2023-10-21T05:30:38.015 INFO:teuthology.orchestra.run.smithi133.stderr:gzip/var/log/ceph/ceph-client.admin.40755.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51796.log 2023-10-21T05:30:38.016 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40755.log.gz 2023-10-21T05:30:38.016 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46369.log 2023-10-21T05:30:38.016 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41377.log.gz 2023-10-21T05:30:38.017 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45488.log 2023-10-21T05:30:38.017 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51796.log.gz 2023-10-21T05:30:38.017 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46461.log 2023-10-21T05:30:38.017 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46369.log.gz 2023-10-21T05:30:38.018 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51367.log 2023-10-21T05:30:38.018 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45488.log.gz 2023-10-21T05:30:38.018 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46461.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.46461.log.gz 2023-10-21T05:30:38.019 INFO:teuthology.orchestra.run.smithi133.stderr: /var/log/ceph/ceph-client.admin.43044.log 2023-10-21T05:30:38.019 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39320.log 2023-10-21T05:30:38.019 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51367.log.gz 2023-10-21T05:30:38.019 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45611.log 2023-10-21T05:30:38.020 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43044.log.gz 2023-10-21T05:30:38.020 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39320.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.39320.log.gz 2023-10-21T05:30:38.020 INFO:teuthology.orchestra.run.smithi133.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50573.log 2023-10-21T05:30:38.020 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47070.log 2023-10-21T05:30:38.021 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45611.log.gz 2023-10-21T05:30:38.021 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50573.log.gz 2023-10-21T05:30:38.021 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44545.log 2023-10-21T05:30:38.022 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51348.log 2023-10-21T05:30:38.022 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47070.log.gz 2023-10-21T05:30:38.022 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44545.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44545.log.gz 2023-10-21T05:30:38.022 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47132.log 2023-10-21T05:30:38.023 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52533.log 2023-10-21T05:30:38.023 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51348.log.gz 2023-10-21T05:30:38.023 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43381.log 2023-10-21T05:30:38.024 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47132.log.gz 2023-10-21T05:30:38.024 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52533.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.52533.log.gz -5 2023-10-21T05:30:38.024 INFO:teuthology.orchestra.run.smithi133.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50119.log 2023-10-21T05:30:38.024 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52056.log 2023-10-21T05:30:38.025 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43381.log.gz 2023-10-21T05:30:38.025 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34613.log 2023-10-21T05:30:38.025 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50119.log.gz 2023-10-21T05:30:38.025 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52056.log.gz 2023-10-21T05:30:38.026 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40704.log 2023-10-21T05:30:38.026 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.34613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34613.log.gz 2023-10-21T05:30:38.026 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48596.log 2023-10-21T05:30:38.027 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49449.log 2023-10-21T05:30:38.027 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40704.log.gz 2023-10-21T05:30:38.027 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47498.log 2023-10-21T05:30:38.028 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48596.log.gz 2023-10-21T05:30:38.028 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39536.log 2023-10-21T05:30:38.028 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49449.log.gz 2023-10-21T05:30:38.028 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39654.log 2023-10-21T05:30:38.029 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47498.log.gz 2023-10-21T05:30:38.029 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48836.log 2023-10-21T05:30:38.029 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39536.log.gz 2023-10-21T05:30:38.029 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41530.log 2023-10-21T05:30:38.030 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39654.log.gz 2023-10-21T05:30:38.030 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48836.log.gz 2023-10-21T05:30:38.030 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51978.log 2023-10-21T05:30:38.030 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41198.log 2023-10-21T05:30:38.031 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41530.log.gz 2023-10-21T05:30:38.031 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48015.log 2023-10-21T05:30:38.031 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51978.log.gz 2023-10-21T05:30:38.031 INFO:teuthology.orchestra.run.smithi133.stderr:gzip/var/log/ceph/ceph-client.admin.41198.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43728.log 2023-10-21T05:30:38.032 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53412.log 2023-10-21T05:30:38.032 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48015.log.gz 2023-10-21T05:30:38.032 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42053.log 2023-10-21T05:30:38.033 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43728.log.gz 2023-10-21T05:30:38.033 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46218.log 2023-10-21T05:30:38.033 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53412.log.gz 2023-10-21T05:30:38.033 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45704.log 2023-10-21T05:30:38.034 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42053.log.gz 2023-10-21T05:30:38.034 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46218.log.gz 2023-10-21T05:30:38.034 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47651.log 2023-10-21T05:30:38.035 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42359.log/var/log/ceph/ceph-client.admin.45704.log: 2023-10-21T05:30:38.035 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45704.log.gz 2023-10-21T05:30:38.035 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51609.log 2023-10-21T05:30:38.035 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47651.log.gz 2023-10-21T05:30:38.036 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42359.log.gz 2023-10-21T05:30:38.036 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39411.log 2023-10-21T05:30:38.036 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45125.log 2023-10-21T05:30:38.036 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51609.log.gz 2023-10-21T05:30:38.037 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50180.log 2023-10-21T05:30:38.037 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39411.log.gz 2023-10-21T05:30:38.037 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45125.log.gz 2023-10-21T05:30:38.037 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53953.log 2023-10-21T05:30:38.038 INFO:teuthology.orchestra.run.smithi133.stderr: 29.2% -- replaced with /var/log/ceph/ceph-client.admin.41198.log.gz 2023-10-21T05:30:38.038 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46431.log 2023-10-21T05:30:38.038 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50180.log.gz 2023-10-21T05:30:38.039 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44975.log 2023-10-21T05:30:38.039 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53953.log.gz 2023-10-21T05:30:38.039 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46431.log.gz 2023-10-21T05:30:38.039 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44757.log 2023-10-21T05:30:38.040 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52426.log 2023-10-21T05:30:38.040 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44975.log.gz 2023-10-21T05:30:38.040 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50057.log 2023-10-21T05:30:38.041 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44757.log.gz 2023-10-21T05:30:38.041 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52426.log.gz 2023-10-21T05:30:38.041 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43412.log 2023-10-21T05:30:38.041 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2023-10-21T05:30:38.042 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50057.log.gz 2023-10-21T05:30:38.042 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48106.log 2023-10-21T05:30:38.042 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43412.log.gz 2023-10-21T05:30:38.043 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47436.log 2023-10-21T05:30:38.043 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2023-10-21T05:30:38.043 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44727.log 2023-10-21T05:30:38.043 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48106.log.gz 2023-10-21T05:30:38.044 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39465.log 2023-10-21T05:30:38.044 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47436.log.gz 2023-10-21T05:30:38.044 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44221.log 2023-10-21T05:30:38.045 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44727.log.gz 2023-10-21T05:30:38.045 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43977.log 2023-10-21T05:30:38.045 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39465.log.gz 2023-10-21T05:30:38.045 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43208.log 2023-10-21T05:30:38.046 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44221.log.gz 2023-10-21T05:30:38.046 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43977.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.43977.log.gz 2023-10-21T05:30:38.046 INFO:teuthology.orchestra.run.smithi133.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43104.log 2023-10-21T05:30:38.047 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51440.log 2023-10-21T05:30:38.047 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43208.log.gz 2023-10-21T05:30:38.047 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52004.log 2023-10-21T05:30:38.047 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43104.log.gz 2023-10-21T05:30:38.048 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47222.log 2023-10-21T05:30:38.048 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51440.log.gz 2023-10-21T05:30:38.048 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38854.log 2023-10-21T05:30:38.049 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52004.log.gz 2023-10-21T05:30:38.049 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46309.log 2023-10-21T05:30:38.049 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47222.log.gz 2023-10-21T05:30:38.049 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37522.log 2023-10-21T05:30:38.050 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38854.log.gz 2023-10-21T05:30:38.050 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42965.log 2023-10-21T05:30:38.050 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46309.log.gz 2023-10-21T05:30:38.051 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.37522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43810.log 2023-10-21T05:30:38.051 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37522.log.gz 2023-10-21T05:30:38.051 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39556.log 2023-10-21T05:30:38.051 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42965.log.gz 2023-10-21T05:30:38.052 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39822.log 2023-10-21T05:30:38.052 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43810.log.gz 2023-10-21T05:30:38.052 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40074.log 2023-10-21T05:30:38.053 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39556.log.gz 2023-10-21T05:30:38.053 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53308.log 2023-10-21T05:30:38.053 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39822.log.gz 2023-10-21T05:30:38.053 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42858.log 2023-10-21T05:30:38.054 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40074.log.gz 2023-10-21T05:30:38.054 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48475.log 2023-10-21T05:30:38.054 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53308.log.gz 2023-10-21T05:30:38.055 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39195.log 2023-10-21T05:30:38.055 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42858.log.gz 2023-10-21T05:30:38.055 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53040.log 2023-10-21T05:30:38.055 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48475.log.gz 2023-10-21T05:30:38.056 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42603.log 2023-10-21T05:30:38.056 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39195.log.gz 2023-10-21T05:30:38.056 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38672.log 2023-10-21T05:30:38.056 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53040.log.gz 2023-10-21T05:30:38.057 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52399.log 2023-10-21T05:30:38.057 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42603.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42603.log.gz 2023-10-21T05:30:38.057 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39070.log 2023-10-21T05:30:38.058 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38672.log.gz 2023-10-21T05:30:38.058 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51900.log 2023-10-21T05:30:38.058 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52399.log.gz 2023-10-21T05:30:38.058 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42113.log 2023-10-21T05:30:38.059 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39070.log.gz 2023-10-21T05:30:38.059 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48931.log 2023-10-21T05:30:38.059 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51900.log.gz 2023-10-21T05:30:38.059 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46157.log 2023-10-21T05:30:38.060 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42113.log.gz 2023-10-21T05:30:38.060 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46188.log 2023-10-21T05:30:38.060 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48931.log.gz 2023-10-21T05:30:38.061 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50968.log 2023-10-21T05:30:38.061 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46157.log.gz 2023-10-21T05:30:38.061 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40728.log 2023-10-21T05:30:38.061 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46188.log.gz 2023-10-21T05:30:38.062 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45581.log 2023-10-21T05:30:38.062 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50968.log.gz 2023-10-21T05:30:38.062 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38456.log 2023-10-21T05:30:38.063 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40728.log.gz 2023-10-21T05:30:38.063 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43017.log 2023-10-21T05:30:38.063 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45581.log.gz 2023-10-21T05:30:38.063 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51663.log 2023-10-21T05:30:38.064 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38456.log.gz 2023-10-21T05:30:38.064 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51458.log 2023-10-21T05:30:38.064 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43017.log.gz 2023-10-21T05:30:38.064 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38601.log 2023-10-21T05:30:38.065 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51663.log.gz 2023-10-21T05:30:38.065 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51458.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51458.log.gz -5 2023-10-21T05:30:38.065 INFO:teuthology.orchestra.run.smithi133.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53896.log 2023-10-21T05:30:38.066 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38618.log 2023-10-21T05:30:38.066 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38601.log.gz 2023-10-21T05:30:38.066 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49052.log 2023-10-21T05:30:38.066 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53896.log.gz 2023-10-21T05:30:38.067 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38618.log.gz 2023-10-21T05:30:38.067 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42798.log 2023-10-21T05:30:38.067 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45035.log 0.0% 2023-10-21T05:30:38.068 INFO:teuthology.orchestra.run.smithi133.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49052.log.gz 2023-10-21T05:30:38.068 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50543.log 2023-10-21T05:30:38.068 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42798.log.gz 2023-10-21T05:30:38.068 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45035.log.gzgzip 2023-10-21T05:30:38.069 INFO:teuthology.orchestra.run.smithi133.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47863.log 2023-10-21T05:30:38.069 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54032.log 2023-10-21T05:30:38.069 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50543.log.gz 2023-10-21T05:30:38.069 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49871.log 2023-10-21T05:30:38.070 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47863.log.gz 2023-10-21T05:30:38.070 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41062.log 2023-10-21T05:30:38.070 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54032.log.gz 2023-10-21T05:30:38.071 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52135.log 2023-10-21T05:30:38.071 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49871.log.gz 2023-10-21T05:30:38.071 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40242.log 2023-10-21T05:30:38.071 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41062.log.gz 2023-10-21T05:30:38.072 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50088.log 2023-10-21T05:30:38.072 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52135.log.gz 2023-10-21T05:30:38.072 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.40242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52295.log 2023-10-21T05:30:38.073 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40242.log.gz 2023-10-21T05:30:38.073 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50027.log 2023-10-21T05:30:38.073 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50088.log.gz 2023-10-21T05:30:38.073 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52295.log.gz 2023-10-21T05:30:38.074 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43842.log 2023-10-21T05:30:38.074 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52611.log 2023-10-21T05:30:38.074 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50027.log.gz 2023-10-21T05:30:38.074 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43842.log.gz 2023-10-21T05:30:38.075 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39990.log 2023-10-21T05:30:38.075 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51553.log 2023-10-21T05:30:38.075 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52611.log.gz 2023-10-21T05:30:38.076 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47743.log 2023-10-21T05:30:38.076 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39990.log.gz 2023-10-21T05:30:38.076 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52373.log 2023-10-21T05:30:38.076 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51553.log.gz 2023-10-21T05:30:38.077 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47405.log 2023-10-21T05:30:38.077 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47743.log.gz 2023-10-21T05:30:38.077 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39864.log 2023-10-21T05:30:38.078 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52373.log.gz 2023-10-21T05:30:38.078 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47405.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47405.log.gz 2023-10-21T05:30:38.078 INFO:teuthology.orchestra.run.smithi133.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52932.log 2023-10-21T05:30:38.078 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52347.log 2023-10-21T05:30:38.079 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39864.log.gz 2023-10-21T05:30:38.079 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47040.log 2023-10-21T05:30:38.079 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.52347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43779.log 2023-10-21T05:30:38.080 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52347.log.gz 2023-10-21T05:30:38.080 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39340.log 2023-10-21T05:30:38.080 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47040.log.gz 2023-10-21T05:30:38.080 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53598.log 2023-10-21T05:30:38.081 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43779.log.gz 2023-10-21T05:30:38.081 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.39340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47529.log 2023-10-21T05:30:38.081 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39340.log.gz 2023-10-21T05:30:38.081 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.53598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38834.log 2023-10-21T05:30:38.082 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53598.log.gz 2023-10-21T05:30:38.082 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46491.log 2023-10-21T05:30:38.082 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47529.log.gz 2023-10-21T05:30:38.083 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42633.log 2023-10-21T05:30:38.083 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38834.log.gz 2023-10-21T05:30:38.083 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38746.log 2023-10-21T05:30:38.083 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46491.log.gz 2023-10-21T05:30:38.084 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46768.log 2023-10-21T05:30:38.084 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42633.log.gz 2023-10-21T05:30:38.084 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.38746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41259.log 2023-10-21T05:30:38.085 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38746.log.gz 2023-10-21T05:30:38.085 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.46768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51755.log 2023-10-21T05:30:38.085 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46768.log.gz 2023-10-21T05:30:38.085 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45186.log 2023-10-21T05:30:38.086 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41259.log.gz 2023-10-21T05:30:38.086 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41287.log 2023-10-21T05:30:38.086 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51755.log.gz 2023-10-21T05:30:38.087 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.45186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42486.log 2023-10-21T05:30:38.087 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45186.log.gz 2023-10-21T05:30:38.087 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54377.log 2023-10-21T05:30:38.087 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.41287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41287.log.gz 2023-10-21T05:30:38.088 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42718.log 2023-10-21T05:30:38.088 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42486.log.gz 2023-10-21T05:30:38.088 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50452.log 2023-10-21T05:30:38.089 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54377.log.gz 2023-10-21T05:30:38.089 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47375.log 2023-10-21T05:30:38.089 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42718.log.gz 2023-10-21T05:30:38.089 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44667.log 2023-10-21T05:30:38.090 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50452.log.gz 2023-10-21T05:30:38.090 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.47375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44059.log 2023-10-21T05:30:38.090 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47375.log.gz 2023-10-21T05:30:38.090 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50150.log 2023-10-21T05:30:38.091 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44667.log.gz 2023-10-21T05:30:38.091 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44089.log 2023-10-21T05:30:38.091 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44059.log.gz 2023-10-21T05:30:38.092 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.50150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48137.log 2023-10-21T05:30:38.092 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50150.log.gz 2023-10-21T05:30:38.092 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49237.log 2023-10-21T05:30:38.092 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.44089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44089.log.gz 2023-10-21T05:30:38.093 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43236.log 2023-10-21T05:30:38.093 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48137.log.gz 2023-10-21T05:30:38.093 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.49237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51848.log 2023-10-21T05:30:38.093 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54404.log 2023-10-21T05:30:38.094 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43236.log.gz 2023-10-21T05:30:38.094 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42828.log 2023-10-21T05:30:38.094 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51848.log.gz 2023-10-21T05:30:38.095 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.54404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43074.log 2023-10-21T05:30:38.095 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54404.log.gz 2023-10-21T05:30:38.095 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.42828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48536.log 2023-10-21T05:30:38.095 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42828.log.gz 2023-10-21T05:30:38.096 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.43074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43074.log.gz 2023-10-21T05:30:38.096 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48991.log 2023-10-21T05:30:38.096 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51330.log 2023-10-21T05:30:38.096 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48536.log.gz 2023-10-21T05:30:38.097 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.48991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34554.log 2023-10-21T05:30:38.097 INFO:teuthology.orchestra.run.smithi133.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48991.log.gz 2023-10-21T05:30:38.097 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.51330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51330.log.gz 2023-10-21T05:30:38.097 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.34554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34554.log.gz 2023-10-21T05:30:38.098 INFO:teuthology.orchestra.run.smithi133.stderr: 26.5% -- replaced with /var/log/ceph/ceph-client.admin.52932.log.gz 2023-10-21T05:30:38.098 INFO:teuthology.orchestra.run.smithi133.stderr: 27.6% -- replaced with /var/log/ceph/ceph-client.admin.49237.log.gz 2023-10-21T05:30:38.191 INFO:teuthology.orchestra.run.smithi133.stderr: 89.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-21T05:30:38.219 INFO:teuthology.orchestra.run.smithi133.stderr: 90.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-21T05:30:53.929 INFO:teuthology.orchestra.run.smithi133.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-21T05:31:05.360 INFO:teuthology.orchestra.run.smithi133.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-21T06:03:35.807 INFO:teuthology.orchestra.run.smithi133.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-21T06:16:07.840 INFO:teuthology.orchestra.run.smithi133.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-21T06:16:29.359 INFO:teuthology.orchestra.run.smithi133.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-21T06:16:29.361 INFO:teuthology.orchestra.run.smithi133.stderr: 2023-10-21T06:16:29.362 INFO:teuthology.orchestra.run.smithi133.stderr:real 45m51.769s 2023-10-21T06:16:29.362 INFO:teuthology.orchestra.run.smithi133.stderr:user 41m1.597s 2023-10-21T06:16:29.362 INFO:teuthology.orchestra.run.smithi133.stderr:sys 2m9.241s 2023-10-21T06:16:29.363 INFO:tasks.ceph:Archiving logs... 2023-10-21T06:16:29.365 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/log/ceph to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433358/remote/smithi133/log 2023-10-21T06:16:29.367 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-21T06:18:22.325 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-21T06:18:22.336 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-21T06:18:22.336 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-21T06:18:22.363 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-21T06:18:22.372 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-21T06:18:22.372 DEBUG:teuthology.orchestra.run.smithi133:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-21T06:18:22.479 INFO:teuthology.orchestra.run.smithi133.stdout: remote refid st t when poll reach delay offset jitter 2023-10-21T06:18:22.480 INFO:teuthology.orchestra.run.smithi133.stdout:============================================================================== 2023-10-21T06:18:22.480 INFO:teuthology.orchestra.run.smithi133.stdout:*hv01.front.sepi 44.4.53.2 2 u 121 256 377 0.165 +0.281 0.297 2023-10-21T06:18:22.480 INFO:teuthology.orchestra.run.smithi133.stdout:+hv02.front.sepi 63.231.80.2 3 u 117 256 377 0.214 +3.239 0.578 2023-10-21T06:18:22.480 INFO:teuthology.orchestra.run.smithi133.stdout:+hv03.front.sepi 74.6.168.72 3 u 113 256 377 0.146 -0.310 0.691 2023-10-21T06:18:22.481 INFO:teuthology.orchestra.run.smithi133.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-21T06:18:22.482 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-21T06:18:22.491 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-21T06:18:22.492 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-21T06:18:22.501 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-21T06:18:22.510 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-21T06:18:22.520 INFO:teuthology.task.internal:Duration was 8820.624013 seconds 2023-10-21T06:18:22.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-21T06:18:22.530 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-21T06:18:22.530 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-21T06:18:22.850 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-21T06:18:22.850 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi133.front.sepia.ceph.com 2023-10-21T06:18:22.851 DEBUG:teuthology.orchestra.run.smithi133:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-21T06:18:22.940 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-21T06:18:22.941 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-21T06:18:23.102 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-21T06:18:23.102 DEBUG:teuthology.orchestra.run.smithi133:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-21T06:18:23.644 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-21T06:18:23.656 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-21T06:18:23.656 DEBUG:teuthology.orchestra.run.smithi133:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-21T06:18:23.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-21T06:18:23.749 DEBUG:teuthology.orchestra.run.smithi133:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-21T06:18:23.780 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = core 2023-10-21T06:18:23.807 DEBUG:teuthology.orchestra.run.smithi133:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-21T06:18:23.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T06:18:23.853 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-21T06:18:23.863 INFO:teuthology.task.internal:Transferring archived files... 2023-10-21T06:18:23.865 DEBUG:teuthology.misc:Transferring archived files from smithi133:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433358/remote/smithi133 2023-10-21T06:18:23.865 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-21T06:18:23.917 INFO:teuthology.task.internal:Removing archive directory... 2023-10-21T06:18:23.917 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-21T06:18:23.962 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-21T06:18:23.973 INFO:teuthology.task.internal:Not uploading archives. 2023-10-21T06:18:23.974 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-21T06:18:23.984 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-21T06:18:23.984 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-21T06:18:24.020 INFO:teuthology.orchestra.run.smithi133.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 21 06:18 /home/ubuntu/cephtest 2023-10-21T06:18:24.020 INFO:teuthology.orchestra.run.smithi133.stdout: 673498 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 21 04:01 /home/ubuntu/cephtest/mnt.0 2023-10-21T06:18:24.020 INFO:teuthology.orchestra.run.smithi133.stdout: 673923 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 21 04:02 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-21T06:18:24.021 INFO:teuthology.orchestra.run.smithi133.stdout: 687830 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 21 04:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-21T06:18:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-21T06:18:24.022 INFO:teuthology.orchestra.run.smithi133.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-21T06:18:24.023 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-21T06:18:24.030 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-21T06:18:24.042 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-21T06:18:24.053 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi133 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-21T06:18:24.053 INFO:teuthology.nuke:Checking targets against current locks 2023-10-21T06:18:24.085 DEBUG:teuthology.nuke:shortname: smithi133 2023-10-21T06:18:24.086 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-21T06:18:24.110 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-20_15:15:00-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7433358', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-21 03:42:57.511951', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKCG8FDeKVCWmI0H1Beeo5dLHysF2o12xGZTrAo6X62dqtrOYeAG7KNJ4hjz6j0NnLRBw4zzerLLkY+JHjo+8w8='} 2023-10-21T06:18:24.161 INFO:teuthology.orchestra.console.smithi133:Power off 2023-10-21T06:18:24.162 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-21T06:18:24.185 DEBUG:teuthology.orchestra.console.smithi133:power off output: Chassis Power Control: Down/Off 2023-10-21T06:18:24.185 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:18:24.209 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2023-10-21T06:18:28.210 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:18:28.320 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2023-10-21T06:18:32.322 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:18:32.432 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2023-10-21T06:18:36.434 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T06:18:36.543 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is off 2023-10-21T06:18:36.645 INFO:teuthology.orchestra.console.smithi133:Power off completed 2023-10-21T06:18:36.751 INFO:teuthology.run:Summary data: description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 8820.624012708664 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi133 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_rraja@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3dd87c3216304c02862d96ba3bd2d91b status: fail success: false 2023-10-21T06:18:36.751 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T06:18:36.832 INFO:teuthology.run:FAIL